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Introduction 


The Quality Elementary Science Teaching (QUEST) program is designed to deliver high-quality 
professional development (PD) in science content to elementary school teachers. QUEST includes both 
content and pedagogical instruction combined with a controlled teaching experience situated within a larger 
professional development context. The design of the PD is meant to provide meaningful and sustained 
changes in teachers’ knowledge and practice, in turn leading to improvements in student learning. Through 
better understanding of how various settings for teacher learning give rise to different kinds of knowing, the 
aim of the QuEST project is to develop a scalable model for implementing situated PD. 


The overarching goals of this four-year National Science Foundation—funded Discovery Research K— 
12 project include the following: 


1. Implement a high-quality situated PD model for K—6 teachers in science. 


2. Conduct a comprehensive and rigorous program of research to study the impacts of this model 
on teacher and student learning. 


3. Disseminate project outcomes to a variety of stakeholders to produce broader impacts. 


Description of the QUEST Project 


The QuEST project runs through three yearlong cycles of PD, each starting with a summer institute, 
from summer 2014 through spring 2017. Each summer institute has had a different physical science content 
focus (magnetism, electrical circuits, properties of matter). For the research design, a total of 28 initial schools 
were recruited from across 7 school districts in Missouri. Within each district, participating schools were 
randomly assigned to conditions, including Treatment 1 (controlled teaching experience PD), Treatment 2 (no 
controlled teaching experience PD), and a group of comparison schools. The 2016 summer institute was 
attended by fifth-grade teachers from 19 of the Treatment 1 and 2 schools across the original 7 school districts. 
Teachers from the comparison schools were offered a two-day summer workshop in July of 2016. 


The design of the summer institute each year is intended to investigate the effects of a controlled 
teaching experience that occurs during the second week of the two-week institute. In 2016, both Treatment 1 
and 2 groups participated in the same Week 1 activities of the summer institute, including the focus science 
content knowledge and introductions to the use of the 5E model of instruction and Universal Design for 
Learning (UDL) in the classroom. However, in Week 2 only the Treatment 1 group participated in the 
controlled teaching experience, implementing the lessons and practices they learned in Week 1 with students 
who enrolled in the Kids QuEST program. In order to ensure both groups received equivalent hours of PD, 
the Treatment 2 group participated in similar activities during Week 2, focused on applying what they learned 
in Week 1 to designing instruction—but those activities did not extend to actual implementation with 
students. Treatment 2 teachers engaged in identifying key concepts in the Missouri Grade Level Expectations 
(GLEs); discussing professional literature related to the 5E, formative assessment, and UDL; critically 
evaluating sample lessons in terms of content accuracy and the appropriateness of instruction and assessment; 
and, using UDL as a framework, adapting activities and assessments for a wide range of learner interests, 
needs, and abilities. To ensure the main difference between the treatment groups was the controlled teaching 
experience, all teacher participants were invited to attend four full-day Saturday follow-up sessions during the 
academic year. 
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QUEST Evaluation Design 


The Educational Policy Improvement Center (EPIC) is serving as the grant evaluator, focusing on 
the meta-level functioning of QuEST and the progress made toward achieving its three overarching goals: 
implementation of a high-quality situated PD model (Goal 1), comprehensive rigorous research on the 
model’s impacts (Goal 2), and dissemination of project outcomes (Goal 3). The first step in the evaluation 
included a meeting with the project staff in November of 2013 to review project timelines, strategies, and 
milestones, serving to provide structure to the evaluator’s final plans for monitoring and assessing the project. 
Year 1 evaluation activities also included instrument development, Advisory Board attendance, and coordination 
with the project researcher on the research design. Years 2 and 3 evaluation activities included evaluation of the 
summer institutes and follow-up PD sessions, and review of school-level achievement data. Year 4 evaluation 
activities included all activities in Years 2 and 3 plus presentation of evaluation activities to the Advisory Board. 


Year 4 Evaluation Activities 


This report focuses on the evaluation activities in Year 4 of the grant, which began with the 
observations of the 2016 summer institute and concludes with the end-of-year teacher survey in April and 
May 2017. 


Professional Development Implementation (Goal 1) 


The evaluation question for this component asks to what extent the QuEST project has achieved its 
goal within the expected timetables using the stated principles and processes (summer PD, situated PD 
experience, quarterly follow-up sessions). EPIC evaluators examined the factors that facilitate or hinder 
implementation plans and the ways in which the project staff managed and overcame barriers. To determine 
the extent to which the PD implementation achieved its goals, the following data collection efforts occurred: 


e Observations of Summer PD Events: The evaluation team observed almost all days of professional 
development during the summer institute of 2016. Researchers modified and used an observation 
protocol based on Learning Forward’s Standards for Professional Learning’ to determine the extent 
to which the PD sessions incorporated the standards of best practice in teacher professional 
development. EPIC evaluators used the structured protocol on best practices in teacher professional 
development across a total of 18 QuEST professional development sessions held during the weeks 
of July 11-22, 2016. A session here is defined as either a morning period of time or afternoon 
period of time, in which a particular learning activity took place. The observations occurred on all 
days of the first week of the institute and on days 1-4 of the second week. 

e Teacher Focus Group Interviews: EPIC evaluators conducted focus groups with teacher participants 
in the QuEST summer institute during the second week of the 2016 summer institute. To gain a 
deeper understanding of the professional development participant experience, focus groups were 
held with three groups of institute attendees. Teacher focus group participants were selected to 
obtain equitable representation from the districts and schools involved. Focus groups were structured 
based on the participants’ role in the QuEST project: Treatment 1 (situated PD treatment) members, 
Treatment 2 (content training but no situated PD treatment) members, and preservice teachers. All 
focus groups were held between July 20 and 22, 2016, and conducted by an EPIC evaluation team 
member. EPIC evaluation team members developed a focus group protocol designed to achieve the 
following: 

oO Gauge teacher satisfaction with professional learning. 

o Determine whether teachers mastered new knowledge and skills. 

o Gather teacher perceptions on how applicable the new knowledge and skills will be to their 
classroom. 


! See http://learningforward.org/standards#. VaiCS7cS2be 
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Focus group sessions were audio recorded. All participants provided oral consent to participate and 
be recorded. Audio recordings were transcribed, and responses then were organized and thematically 
analyzed using NVivo qualitative software. Once themes were identified, data were summarized by 
participant category. 

Project Stakeholder Interviews: Following the summer institute, EPIC evaluation team membets 
conducted debriefing sessions with groups of individuals involved in the institute. The sessions were 
conducted via WebEx conferencing. Debriefing sessions were structured by the role that individuals 
played during the summer institute: principal investigators, the faculty/teacher instructional staff, 
graduate research assistants (GRAs), and the project support staff. Individuals were asked to provide 
feedback on the summer institute in terms of what went well, what challenges were encountered, and 
recommendations for future institutes. Recordings of the debriefing sessions were summarized across 
and within participant categories. 

Observations of Quarterly PD Follow-up Events: EPIC evaluators attended one of the four 
Saturday follow-up sessions in academic year 2016-17, on April 8, 2017. The same observation 
protocol used in the summer institute was used to provide feedback on best practices in teacher PD. 
Teacher Survey: A teacher survey was developed in conjunction with the QuEST staff and 
administered during the final follow-up PD session on April 8, 2017, to assess the outcomes of 
participation in the QuEST project, including perceived impact on teacher science content 
knowledge, pedagogical content knowledge, and classroom practice. 


Other ongoing activities that occurred throughout the year include the following: 


o Review of QuEST project artifacts (e.g., reports, internal communications regarding project 
progress, and agendas and minutes of key planning meetings) 

o Review of QuEST products (e.g., PD materials, ConcepTests, and other PD resources) 

o Regular check-in meetings with project principal investigators 


Research (Goal 2) 


The evaluation questions for this component ask to what extent the QuEST project research phases 


are implemented as proposed and to what extent the implemented research plan has answered the four 
identified research questions. Evaluators examined factors that facilitate or hinder the research program and 
how the project staff managed and overcame barriers. Data collection activities included the following: 


Project Stakeholder Communication: A combination of formal interviews and regular informal 
phone conference meetings was used to check on progress toward project goals throughout the year. 
Advisory Board Meeting: EPIC evaluators attended and presented at the final advisory board 
meeting on May 18, 2017. 

Review of QuEST Project Artifacts: The QuEST staff and EPIC evaluators both had access to 
and used the project management tool Asana in order to share and review relevant assessment tools 
and the data associated with the development and implementation of the tools. 

Proximal Data Review: EPIC evaluators reviewed project research data collected by the QUEST 
project staff on an ongoing basis, both at the teacher level (i.e., teacher data from MOSART, ULC 
instrument, UDL test, observation protocols, lesson plan tasks/ interviews, CoRE and PaPeRs data) 
and the student level (i.e., ConcepTest results). The purposes of these reviews include spot-checking 
research findings, assessing reliability of observations, and incorporating the project research data 
into the evaluation data sets to assess progtess toward objectives. 

Distal Data Review: EPIC evaluators worked in conjunction with the QuEST staff to retrieve and 
review the analysis of Missouri Assessment Program (MAP) school- and grade-level results in science 
(Grade 5), mathematics, and reading. (Note: In school year 2013-14, a revised assessment system was 
implemented in Missouri, aligning with the Smarter Balanced test design and blueprints.) 
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Dissemination (Goal 3) 


The evaluation question for this component asks to what extent the project team is successful in 
disseminating the QuEST model and findings. Evaluation includes a review of project reports, manuscripts, 
articles, presentations, doctoral dissertations, and products (e.g., PD materials, assessments, and other 
resources). Summer institute attendees are asked to complete a follow-up survey the spring following their 
participation to assess the extent to which they have used materials and new learning in their professional 
development activities. Evaluators also will assess the rigor and reach of publication or presentation venues 
(e.g., peer-reviewed scholarly journals, practitioner-focused publications, and peer-reviewed presentations). 
Interviews with key stakeholders, journal impact indices, journal subscription figures, presentation and 
colloquia audience attendance, and reviews of relevant research data are used to determine the extent to 
which the project is likely to be sustainable, scalable, and of high impact. 


Summative Evaluation Activities 


Project evaluation activities in the upcoming year will include a summative review and analysis of 
project data from all years, across the evaluation questions, summarized as follows: 


Professional Development Implementation (Goal 1) 


e Observations of all summer institutes, summer 2014-2016 
e Teacher focus group interviews, summer 2014-2016 

e §=©Project stakeholder interviews, summer 2014-2016 

e Observations of quarterly PD follow-up events, 2014-2017 
e Teacher surveys, spring 2015-2017 


® Observation of extended project activities, Summer Writer’s Retreat, and QUEST PD for comparison 
group, 2017 


Research (Goal 2) 


e Project stakeholder interviews, summer 2014-2016 

e =Review of QuEST project artifacts, 2014-2017 

e Proximal data review, teacher and student levels, 2014-2017 
e Distal data review, student level, 2014-2017 


Dissemination (Goal 3) 


e Final summative review of project reports, manuscripts, articles, presentations, doctoral dissertations, 
and products (e.g., PD materials, assessments, and other resources), 2014-2017 


e Teacher surveys, spring 2015-2017 
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Year 4 Evaluation Findings 


This section details the findings from the evaluation of the fourth year of the QUEST DRK~-12 grant, 
starting in May 2016 through June 2017. 


Professional Development Implementation Evaluation 


This section provides a summary of the findings from the evaluation activities related to the QUEST 
professional development implementation. 


Summer Institute 2016 


The evaluation data collected at or immediately following the 2016 summer institute include 
observations, focus groups, and interviews. 


Week | Professional Development Observations 


The evaluation team began observing on Day 1 of the first week of the institute. Morning 
observations included all institute participants engaging with the science content learning curriculum. The 
afternoon sessions Monday through Thursday included all institute participants in sessions focused on 
instructional design and pedagogy, including Universal Design for Learning, the 5E model, and the use of 
conceptual storylines in lesson planning. The afternoon session on Day 5 began with all institute participants, 
then members of the Treatment 1 and Treatment 2 teams separated into their working groups to begin 
planning for Week 2. Treatment 1 teachers and preservice teachers were tasked with lesson planning for the 
first day of Kids QuEST. Treatment 2 teachers were asked to think about resources they might want to bring 
back with them, specific to their school/lessons, to use in their lesson analysis and planning in Week 2. 
During the morning and afternoon sessions, facilitators were on hand at all times to present materials, answer 
questions, and circulate among the participants. Whole class, small group, and individual work was observed 
within each session. The overall level of participant involvement was high (more than 75% on task 
throughout the observation). 


The following tables summarize the observations conducted by the evaluation team. Table 1 includes 
observed professional development standards from Week 1, separated by morning and afternoon sessions. 
Table 2 includes observed professional development standards from Week 2, separated by Treatment 1 and 
Treatment 2 sessions. Given the continuous nature and design of the two-week institute, individual standards 
and/or indicators may not have been obsetved within every session; the tables summarize whether or not 
each standard/indicator was observed at all during the observation period. 


Table 1. Summary of Observed Professional Development Standards — Week 1 


Standard/Indicator | AM Session PM Session 


Selection and Organization of Content 


Clearly stated the purpose of the instruction 


Defined the target audience and necessary prerequisite skills 


Clearly defined expected outcomes 


Formulated a limited number of goals and/or objectives 


Developed a conceptual framework to highlight major ideas to be presented 
and to organize the content 
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Standard/Indicator | AM Session PM Session 


Reseatch 


Described the research, evidence, theory, or reports from practitioners that 
supported the content 


Avoided conjecture and lots of personal opinion, but included valid alternate 
opinions to help the participants make their own judgments 


Fully cited all references and research, where and as appropriate 


Identified professional standards or content standards where applicable 
(e.g., CCSS, NGSS, GLES) 


When possible, cited evidence of effectiveness of practices presented in the 
training or presentation 


Principles of Adult Learning 


Drew upon and honored learners’ prior knowledge and experience 


Provided opportunities for learners to connect new learning to their own work 


Designed and used problem-centered activities 


Provided strategies, tools, and techniques that participants could begin to use 
immediately 


Provided 5E model tools to target teachers’ difficulties in implementing the 5E 
model 


Provided formative assessment strategy training to increase teachers’ 
assessment literacy 


Discussed the concept of seamless assessment strategies and how to align 
them to the 5E model 


Provided Universal Design instruction and strategies for use to increase 
teachers’ knowledge of UDL principles to use with diverse learners 


Offered opportunities for participants to assume responsibility for their own 
learning (i.e., to be as self-directed as appropriate given content) 


Community of Learners 


Encouraged dialogue and sharing among participants 


Encouraged dialogue and sharing between participants and PD staff/content 
experts 


Incorporated opportunities for team building and collaboration during sessions Pe) e 
and in designed follow-up 


Considered structures and practices that connect participants across time 


Encouraged risk taking @ @ 


Provided for celebration of small successes and learning from failures 


Practice 


Incorporated opportunities for participants to practice new skills in a safe 
workshop environment and to receive feedback from facilitator and colleagues 


Designed follow-up activities @ 
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Standard/Indicator | AM Session PM Session 
Included structures to track participants following the professional @ 
development 


Encouraged peer classroom observations when appropriate 


Developed capacity of individuals at site to provide leadership for follow- 
through 


Diversity 


Established norms of respect, openness, and listening through both the content 
and process 


Used language and media elements that were respectful of and appropriate to 
all 


Planned for an inclusive approach (i.e., used strategies to engage all 
individuals) 


Provided opportunities for educators to share knowledge, skills, and strategies 
for involving families or other stakeholders appropriately 


Process Design 


Incorporated a variety of presentation strategies and activity formats 


Included media elements that appropriately support the content 


Ensured that participants have opportunities to develop or review related 
knowledge base 


Varied learning activities, interspersing didactic with active, hands-on learning 


Designed for all learning styles: the visual, auditory, kinesthetic 


Structured daily activities and kept on task so that adequate time was available 
for instruction, activities, and reflection 


Self-Assessment 


Ensured that participants were clear about expectations for their own learning 
and change 


Allowed for practice and assessment of all stated outcomes 


Encouraged participants to identify and build on their individual strengths 


Provided opportunities for participants to identify barriers to change 


Evaluation 


Built in debriefs at critical points in the training (e.g., at the end of a day) 


Sought participant feedback throughout the formal learning experience — both 
formally (e.g., quick feedbacks at day’s end) and informally (e.g., by asking @ @ 
participants to signal how it’s going for them at intervals throughout the 
learning experience) 


Scheduled ample time for end-of-session evaluation Ce) @ 
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Week 2 Professional Development Observations 


On Days 1 and 2 of the second week of observations, one EPIC evaluator observed Treatment 2 
teachers in the lesson analysis/planning sessions throughout the day. The other evaluator divided time in each 
of the Kids QuEST rooms, observing the implementation of the Kids QuEST in the mornings and in 
Treatment 1 teachers’ planning sessions that occurred each afternoon. 


Kids QuEST (Treatment 1). In the mornings, small teams of Treatment 1 teacher and preservice 
teacher participants taught their newly developed properties of matter lessons to groups of local students who 
had registered for the Kids QUEST camp. During the morning Kids QuEST class, facilitators and staff 
members were available but did not participate in the instruction. They observed the instruction in different 
classrooms and provided support and materials to their assigned groups as needed. In the afternoons, the 
teams of teacher participants reflected on the morning instruction and planned instruction for the next day. 


Lesson Analysis and Planning (Treatment 2). The full-day sessions in Week 2 were designed to 
allow time for Treatment 2 teacher participants to dig deeper into the materials learned in Week 1. The goal 
of Week 2 was to analyze the science content curriculum from Week 1 and design fully developed lessons for 
their own science units. Three facilitators were in attendance. Participants worked in school/district groups 
with the science curriculum and materials from Week 1. All facilitators circulated throughout the entire 
session, observing participant progress, posing and answering questions, and providing support and materials 
as needed. During the session, evaluators observed whole class, small group, and individual work. The overall 
level of participant involvement was high (more than 75% on task throughout the observation). 


Table 2. Summary of Observed Professional Development Standards — Week 2 


Situated — Lesson Analysis & 


Standard/Indicator PD (Tx1) Planning (Tx2) 


Selection and Organization of Content 


Clearly stated the purpose of the instruction 


Defined the target audience and necessary prerequisite skills 


Clearly defined expected outcomes 


Formulated a limited number of goals and/or objectives 


Developed a conceptual framework to highlight major ideas to be 
presented and to organize the content 


Reseatch 


Described the research, evidence, theory, or reports from practitioners that 
supported the content 


Avoided conjecture and lots of personal opinion, but included valid ®@ 
alternate opinions to help the participants make their own judgments 


Fully cited all references and research, where and as appropriate 


Identified professional standards or content standards where applicable @ 
(e.g., CCSS, NGSS, GLES) 


When possible, cited evidence of effectiveness of practices presented in the 
training or presentation 


Principles of Adult Learning 


Drew upon and honored learners’ prior knowledge and experience 


Provided opportunities for learners to connect new learning to their own 
work 
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Situated — Lesson Analysis & 

Standard/Indicator PD (Tx1) Planning (Tx2) 
Designed and used problem-centered activities ) @ 
Provided strategies, tools, and techniques that participants could begin to @ e 
use immediately 
Provided 5E model tools to target teachers’ difficulties in implementing the ®@ e 
5E model 
Provided formative assessment strategy training to increase teachers’ @ @ 
assessment literacy 
Discussed the concept of seamless assessment strategies and how to align @ @ 
them to the 5E model 
Provided Universal Design instruction and strategies for use to increase @ @ 
teachers’ knowledge of UDL principles to use with diverse learners 
Offered opportunities for participants to assume responsibility for their @ @ 
own learning (i.e., to be as self-directed as appropriate given content) 
Community of Learners 
Encouraged dialogue and sharing among participants © oO 
Encouraged dialogue and sharing between participants and PD staff/ @ @ 
content experts 
Incorporated opportunities for team building and collaboration during @ e 
sessions and in designed follow-up 
Considered structures and practices that connect participants across time @ 
Encouraged risk taking ©) @ 
Provided for celebration of small successes and learning from failures @ © 
Practice 
Incorporated opportunities for participants to practice new skills in a safe 
workshop environment and to receive feedback from facilitator and 
colleagues 
Designed follow-up activities 
Included structures to track participants following the professional @ @ 
development 
Encouraged peer classroom observations when appropriate 
Developed capacity of individuals at site to provide leadership for follow- @ @ 
through 
Diversity 
Established norms of respect, openness, and listening through both the e 
content and process 
Used language and media elements that were respectful of and @ @ 
appropriate to all 
Planned for an inclusive approach (i.e., used strategies to engage all @ ® 
individuals) 
Provided opportunities for educators to share knowledge, skills, and @ e 


strategies for involving families or other stakeholders appropriately 
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Situated — Lesson Analysis & 


Standard/Indicator PD (Tx1) Planning (Tx2) 


Process Design 


Incorporated a variety of presentation strategies and activity formats 


Included media elements that appropriately support the content 


Ensured that participants have opportunities to develop or review related 
knowledge base 

Varied learning activities, interspersing didactic with active, hands-on 
learning 


Designed for all learning styles: the visual, auditory, kinesthetic 


Structured daily activities and kept on task so that adequate time was 
available for instruction, activities, and reflection 


Self-Assessment 


Ensured that participants were clear about expectations for their own 
learning and change 


Allowed for practice and assessment of all stated outcomes 


Encouraged participants to identify and build on their individual strengths 


Provided opportunities for participants to identify barriers to change 


Evaluation 


Built in debriefs at critical points in the training (e.g., at the end of a day) 


Sought participant feedback throughout the formal learning experience — 
both formally (e.g., quick feedbacks at day’s end) and informally (e.g., by @ @ 
asking participants to signal how it’s going for them at intervals 
throughout the learning experience) 


Scheduled ample time for end-of-session evaluation O oO 


Teacher Focus Groups 


Focus group members were asked to provide feedback about the summer institute in terms of the 
following themes: a) communication of objectives, b) organization and delivery of material, c) facilitator 
qualities, d) applicability to classroom instruction, and e) experience with the professional development. This 
last category was modified depending on the team to which the focus group member belonged. Additionally, 
focus group members were asked to consider and provide separate answers to the prompts in each category 
based on the week of the institute (ie., Week 1 activities versus Week 2 activities). 


Treatment 1 (Situated PD) 


Objectives. Treatment 1 teachers varied in their understanding of the summer institute objectives 
ptior to Week 1. Some Treatment 1 teachers heard about QuEST through an administrator who also 
provided links to additional information, such as QuEST program reviews. Other teachers shared details they 
learned from colleagues who previously attended QuEST. Most teachers had a basic understanding of 
QuEST project logistics, but were unsure of what exactly they would do. Many focus group members knew 
that the institute would entail lesson planning and learning about science and matter, but acknowledged they 
were not aware of the teaching component in Week 2. Teachers also noted they were surprised by the 
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intensity of the 5E Learning Model and UDL educational frameworks that were covered during the summer 
institute. They wished that they had been informed that these topics would be explored in depth so that they 
could have prepared before arriving at the institute. Others mentioned that their lack of prior knowledge 
about UDL made them mote enthusiastic and motivated to participate in QuEST, because they recognized 
the opportunity to learn something new. All Treatment 1 teachers understood the objectives of Week 1 once 
it began. 


Materials. Treatment 1 teachers appreciated the materials they received in their teacher kits and 
affirmed that they were high quality. As one teacher who had participated in QuEST before conveyed, 
“They’ve always given quality materials as far as our tubs go. It’s like they think outside the box... . Who 
would have known that there was some such thing as density rods or density blocks and all that to be 
beneficial in our classrooms?” Other teachers valued the curriculum and its layout, which includes guiding 
questions, noting how these materials informed the questions they posed to their students as well. 


Delivery and Organization. The hands-on nature of the morning sessions and the ability to be in a 
learner’s role was cited by many Treatment 1 teachers as the most beneficial aspect of QUEST. As one 
teacher stated, “I felt like I learned a lot, and also me just being a learner, it’s going to help me make the 
lessons for my kids. It makes me feel more confident, and also being prepared. One of the reasons I haven’t 
liked science is because you always have a kid or two who will ask me a question I don’t know. Just 
experiencing it as a student helps with that. I feel like since I understand it deeper, then I’m not as nervous 
when I get those questions because we can just start with what we know and break it down.” Another teacher 
reported, “When you go to PD all the time, you never get those opportunities. Usually you’re sitting there, 
and you’re listening to somebody feed you all this information. You never get the time to actually do it.” 
Other teachers spoke of how the group structure was differentiated: “I really liked how we got to work with 
our same team, starting every morning. Because then we got to build relationships within that team, and then 
we had time to meet other people, too, throughout.” Several Treatment 1 teachers did note how full the day 
felt and reported feeling “saturated.” They suggested either shortening the workday, condensing the 
information, or breaking up the activities. 


Facilitators. Many Treatment 1 teachers appreciated the different areas of expertise represented by 
the QuEST staff, ranging from experts in science instruction and special education to those working on their 
doctorate in physics. Several teachers referenced the cohesiveness of the facilitators collectively and their 
openness in incorporating teacher feedback into the institute. As one teacher stated, “My overall feeling with 
this whole experience is that we’re all in it together, that there’s no gulf between instructors or facilitators, and 
we as students. That everybody is in this equally together.” The willingness of the facilitators to help guide the 
teachers through any confusion made the participants feel comfortable asking questions and taking risks. 


Applicability. Several Treatment 1 teachers described the usefulness of the step-by-step format of 
the organization of the 5E. “In the classrooms that I’ve worked in, and the kids I’ve had encounters with, that 
seems to be the most important, that you’re really establishing that structure and that organization with 
them.” Another teacher described connections between UDL and differentiation: “There are great things to 
get from the differentiation standpoint, then there’s great things to get from the UDL standpoint. I’m going 
to figure out how to incorporate both of them in my classroom because I| think there’s great things about 
both, then there’s things that I may not use from, as well.” 


Preparation. Treatment 1 teachers felt prepared to implement UDL in their classrooms, noting 
connections to modifications and differentiation of instruction to which they are already accustomed. 
Similarly, several teachers connected their backward planning process with the conceptual storyline. One 
teacher shared appreciation for the conceptual storyline chart as a tool to help organize planning. The primary 
constraint raised by Treatment 1 teachers around the implementation of what they learned during Week 1 was 
a limited amount of scheduled class time, which they felt might not facilitate a pace at which students can 
come to understand concepts. Several teachers raised the need for buy-in or support from principals or 
superintendents. As stated by one teacher, “When you’re constantly being told, ‘We need to get to this and 
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this and this,’ and then to spend one day on an engage [activity] is difficult . .. even though I know that’s 
what they need. That’s how they learn.” 


Professional Development Experience. Treatment 1 teachers differentiated the engaging and 
hands-on nature of QuEST as superior to previous PD they had received. “Unfortunately a lot of PD is 
staying and listening to a little lecture. This is not at all that. It’s way more hands-on.” Another teacher 
described it, “Without the practice of questioning, I don’t think it would have been... . I would maybe have 
remembered, ‘Oh yeah. I should be questioning and not giving them the answers.’ It’s things like that that we 
learned last week, but then practicing them right away this week, it'll stick. P’ll carry [the things that we 
learned] through the year instead of just being told that it’s a good practice.” 


Treatment 1 teachers made the following comments in response to the focus group prompt, Edements 
of this professional development experience from which I learned most: 


I’ve learned the most in our planning conversations that we've had with Delinda and Dante. They’re very 
straightforward with us in saying that this is not an easy concept. This is not something [where]... a light 
bulb’s going to go off immediately, snap-of-the-fingers type thing. It’s something that you're going to have to 
plan for, put into practice, see how it works. What's good, what’s bad, make those changes, and really be 
thinking about it. Not only in each piece of it, but as a whole. How does it function as each piece, then how 
does that function as a whole. 


My biggest learning moment is being able to link the different components of 5E. Where before I'd just done 
a general lesson plan with objectives and steps and an assessment at the end, this, to me, has been a lot 
different. Each part of the lesson plan is almost like a separate component, yet it snaps into the other via a 
linking question, or something of that nature. For me, that’s been the biggest learning point of this two weeks. 


My biggest thing that I took away was I already knew how a 5E [lesson] should look in a classroom, so my 
biggest thing that I took away was the conceptual storyline. You're engaged, it should follow one after another. 
Then in the seamless assessment, you can have different types of assessments after each different phase of the 
SE. 


I think [conceptual storyline] has been the most exciting thing for me. We were in the classroom the whole 
Jirst week, learning about these different things, and then being able to get to apply them, and actually get to 
see it work. Because they tell you about this conceptual storyline, and you're like “Yeah, of course it makes 
sense,” but actually getting to see it. Then when you teach that first lesson, and it’s all about ‘What is 
matteré” Y ou don’t want to give it away, but then our group ended up giving it away in the Engage, and I 
was really angry. | was mad that the conceptual storyline was interrupted, and I wanted to see them get to 
that point, which thankfully a lot of the kids weren't even listening. We were able to question them in that 
Explore to get them thinking differently now. “Ob, maybe that’s not it,” so at least we've gotten away from 
what they heard the definition was, or “what was matter.” That, I think, was the biggest part, was that 
application. Getting to see it all so smoothly, and make sense. 


My general application, the more broad view would be the conceptual storyline to take away. Then also, I was 
really excited to learn a lot of fifth-grade content. Because I feel like, in college, it was more general “Ob, 
you're going to learn science-y things,” and they'd hit a couple big science things, but I have no idea what fifth 
graders were supposed to know or what you would be learning. I was excited to hear even the different 
molecules in water, and what’s between the molecules. I was excited to know more of what Vil be teaching. 


How it all fit together. My brain always thinks like a puzzle, and I like to think about how you put each 
piece of the puzzle together. I thought about when we were starting with the conceptual storyline and how it fit 
into the 5E, and each piece that it got connected. Then I felt like, at the very end, you could step back and 
you could look and it really all fit together. It was really hard to think about from the beginning, when they 
talked about conceptual storyline, and it needed to flow throughout the whole thing. We struggled with that, 
but now that I look back on our four days of teaching, it really did fit together. 


I think the biggest element is to let kids do the thinking without the teachers jumping in and giving the 
answers before they can actually explore. 
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Actually exploring the curriculum that we're going to expect from kids... . I’ve learned matter in school, but 
it’s been so long ago, and we expect them to just pick up on it right away. Kind of refreshing our knowledge 
and pushing us further helps us be able to teach and follow their thinking a little more. 


I think I learned the most from getting the 5E in two different ways, like as a learner and then as a teacher, 
the practice of it. 


I was going to say 5E, but going at the learner’s pace. 


My two are actually exploring it as the learner, and then being able to turn right around and teach it as a 
teacher, that is in planning with the team because it made you question some of the things that you might have 
had missed concepts about from the previous week. It was cleared up and you could then turn around and 
teach it to the kids. I thought that was just awesome. 


I learned the most from being able to practice planning a storyline and then trying to follow that out through 
lessons and teaching it. 


When asked about what elements of the professional development could be improved, the 
Treatment 1 teachers offered the following comments: 


In my opinion, improvement might be in extending the student contact hours a little bit longer each day, so 

we'd have more time to apply some of these things that we're learning with them. I don’t feel like we abvays 
have time. Three hours just goes by really quickly. Maybe four, even just an extra hour would make a big 

difference. 


Going back to say that this is the best PD that I’ve ever been to, and I’ve learned so much. Now, thinking 
about going back [to school], it’s overwhelming because I’m one of those people that I want it to be like this 
all the time, and I know that I’m not going to be able to physically do it. For me...I wish we could have 
those resources, to makes us feel at ease... as far as lesson planning—wise, and on other units. Because you 
give us this one, we're going to nail this one out of the park. 


I think that if I could feel like I was more comfortable beyond this, because I think about going back to my 
school and V'd like to be able to do more of this throughout other science units. I don’t know if I necessarily 
feel prepared enough to be able to develop all of those conceptual storylines along all those other subject areas. I 
wish we would have had turnouts on topic that didn’t have anything to do with matter, and let us just say 
“Here’s a topic, where would you start?” Maybe help us and guide us in that direction, and say, “Here’s 
where we started.” Because I know they spent a lot of hours creating those, and I'm not sure I feel like I'm 
comfortable enough to produce a document that’s that high quality. That’s what I'd like to be able to do, but 
I'd like to Rnow a little bit, “How did they do it? What was their process? How did they get to create 
those?” Maybe we could’ve spent an afternoon doing something like that, teaching us more on how to develop 
[conceptual storylines] on other units that we had. 


I would have wanted some more options as to how to teach certain concepts. I feel like we got great ideas, but I 
feel like you realized some of these ideas aren't going to work with the students that you have. Y ou get to a 
point sometimes, as a teacher, where you're like “T’ve exhausted everything in my bag of tricks. What do I do 
now?” Maybe having some other options for certain ones might have been nice. 


What I’m thinking right now is a little more nitpicky for the actual PD part. One of them that stuck out, 
I'm sorry if this is picky, is on the very first day when they were doing the introductions. Me, coming from 
[another state] and not knowing anybody, I hadn’t even met these guys yet. The only concept of OnEST I 
had was the packet that I saw on email and that it was going to be really good. When they did the 
introduction, everyone was introduced like they had to be convinced to come. Because they Rept saying, ‘We 
dragged this person in,” so I was like, “Ts this a program nobody wanted to be in?” You know what I mean? 
I know what it’s like when you're in a community and everyone’s like, “We need help, you have to do this,” 
and you're like, “I don’t really want to put another thing on my plate, but fine, D'll do it.” They Rept saying 
that for everyone, and I was like, “I feel bad. Do they want to be here, or did they just really, really need help 
so they had to drag people?” I think that’s the word that they used, and they Rept saying it. Even just that 
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word. It would be extremely easy for them to pick a different word. It did change my feeling, and how I felt, 
being in the room. 


I would say part of that conceptual storyline. When you go back and look at your curriculum, where do I 
start? How do I know that this is the beginning? I’m not a science expert, so how do I know exactly where to 
start in this? We were looking back on our lesson plan on matter in our science curriculum, and were like, 
‘Why are we starting with this, and are they doing that with every concept they’re teaching in science?” 


The afternoon sessions in... Week 1—making them more relevant. I think we got burnt out pretty quick 
on the UDL, and maybe more options than that. 


Just more hands-on, more action [during the afternoon sessions], more something because we're kind of. . . 
saturated. Reflecting or something different in the afternoon to keep us moving. I mean, I know we're grown 
and we can go to the bathroom or whatever, have free will—but if they would have structured breaks. Like, 
“At this time, we're going to take a break.” [Also] .. . some groups wasn’t [sic] very welcoming of other 
people outside of their school. That kind of make_you feel real small because it’s like, "Maybe you should go 
work with such and such,” because I was by myself, and it was just like putting us together, because we're 
still from the same district. 


With the instructors, like with the planning in the afternoon ... if we need you, can we come and get you... 
because I felt like they were hovering over us a lot. 


I think that sometimes people think when we say we want to move around, it means, “We need to get them 
up, put them with a new group of people, and make them do something else.” That’s not what I like. I feel 
like we almost did that too much. For me, I work best if I can be around the same group of people for a little 
bit. You have to get to know each other a little bit, even if it’s all new people. You need some time. Like our 
morning groups, we were with people we didn’t know. The first day, I was kind of like, “I don’t know,” but 
then it was great because I got to know them, and it was great. Then when you have to do that too much, you 
don’t have time to build the relationships, and you can’t work together. Y ou can’t find your roles. Just more 
hands-on things, but with people that you're comfortable with, either people from your school or people that 
you've already worked with in some other group. Constantly having to have a new group was almost part of 
the exhaustion in itself. It’s work to get to know people, and to have to do it every couple hours, it was hard. 


I think that we could just spend more time on the curriculum, exploring. Maybe three fourths of the day on 
that, and then just a little bit of, “Hey, here’s how you could plan those lessons.” Even if we started with, 
“Here’s how you could plan those lessons,” and then explored the rest of the day. I feel like if you start with 
the more boring stuff, then you can look forward to something throughout the day, maybe just exploration, 
hands-on engagement. 


I dreaded lunch being over because I'm like, “Man, we've got to go to the lecture part.” I looked at it as just 
like a college lecture... . They got us up a few times. I know they didn’t want us to sit in our seats the whole 
time. I don’t think that was their intentions, but it just wasn’t enough. I think it was nice to have a morning 
group... maybe we should have had that PM group the whole week. You're still having other people, but 
that way, just as you said, you're getting to know people and sticking with it. 


I just wish coming into this, I’m going to be a new teacher, and she’s going to be on my team. With being a 
Jirst year teacher, there’s nerves, and I don’t really know what to expect. Just if we were together on a team, 
Just putting the schools together, but still having some interaction with other places .. . like in the afternoon 
maybe. But if there’s a team that could be just your school, I think that would be helpful, but I don’t want it 
to turn into snobby like, “You're not in our group” kind of thing. 


Treatment 2 (No Situated PD) 


Objectives. Leading up to the institute, all but one Treatment 2 teacher was familiar with the 
QuEST project through conversations with other teachers who previously attended a summer institute. 
However, some Treatment 2 teachers did not have a clear idea of the structure for the two-week professional 
development. As a teacher conveyed, “I knew that it was two weeks and about properties of matter and I 
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knew that that was a big weakness of mine so I jumped in. I knew that we were going to be learning and 
teaching, but that’s about it.” All Treatment 2 focus group participants indicated the objectives became clear 
once they began the first week. Several teachers elaborated on a general perceived objective of “building our 
own background knowledge.” As one member conveyed, “They didn’t want us to go too fast, they didn’t 
want to say, ‘okay you already know it, let’s move on.’ It was to get to the underlying idea of . . . the why.” 


Materials. Teachers were excited about the classroom materials they received as part of their 
participation in the summer institute and the ability to bring a set of materials back to their school. All 
Treatment 2 teachers also appreciated the quality of the training materials, as conveyed by one participant: 
“Even though they focused on lessons for adults, they keep the ideas where you could take the same concept 
and materials even and just tweak it to match the fifth-grade plan, which was awesome.” One teacher 
especially noted the flow of the training materials, saying, “The flow of it was really nice, how it was flowing 
from one to the other and it just went together really well.” 


Delivery and Organization. Another teacher raised the idea of flow, while describing the Week 1 
training, saying, “They knew how it was going to flow so when they finished the learning cycle, they knew 
that you could go pick up the next packet. They knew that everybody was going to be in different places at 
different times. It didn’t seem chaotic. .. .” Another Treatment 2 teacher noted the use of purposeful ways to 
keep the group relatively in the same place, while facilitating this flow: “I think they also thought ahead, like 
maybe the group was falling behind. They asked us to do a couple of things, maybe that night, just to keep it 
in a certain area, maybe close enough.” 


Facilitators. Treatment 2 teachers appreciated the QuEST facilitators’ approaches and the strength 
and diversity of their collective expertise. As one teacher conveyed, “They have their specialties, like there is a 
go-to person for a deeper understanding of the teaching, like elementary teaching versus the physics 
concepts.” Other Treatment 2 focus group participants pointed out the strength of the adjustments the 
facilitators made in response to participants’ needs or questions. One teacher stated, “They came in with a 
game plan but also tried to meet our needs and always asked for our opinions and how to better the lesson 
and any idea that afternoon and daily. I think that is really important and I really appreciated that and my 
team did as well.” Another teacher conveyed the effect this attention had, adding, “I really feel like we left 
with the best feedback the next day.” Treatment 2 participants also noted that QuEST facilitators were 
helpful in seeking out additional resources, such as websites, and sharing them with individual teachers. 


Applicability. Treatment 2 teachers agreed all the topics they learned were valuable. Some teachers 
illustrated their agreement by sharing their initial connections to what they currently do in their classroom. 
For example, referencing the 5E and the conceptual storyline, one teacher shared, “We use Pearson science, | 
didn’t even notice it until this week, it’s set up for five weeks .. . but the problem is, it’s not very conceptual. 
It isn’t a flow .. . [but] if we tweak it a bit, it can.” 


Preparation. All Treatment 2 teachers agreed they felt better prepared to implement what they 
learned during Week 1. As one teacher framed the two weeks of PD: “Week 1 was making sure that we knew 
the work and knew what we were teaching and then Week 2 was bringing in the conceptual storyline, UDL, 
and all of [the rest of the Week 1 material]. That makes sense as to why we did that.” Though all teachers 
conveyed that they were better prepared to implement what they learned, school schedules and testing were 
raised as several potential constraints: “Sometimes I think with time constraints, that’s where I find my goals. 
I kept it obvious, when I should have went down underneath.” 


Professional Development Experience. Treatment 2 teachers noted Week 2 as being harder than 
the first week. One teacher mentioned getting into the flow of lesson writing and needing to jump to other 
things such as the conceptual storyline: “I think it was harder for us because we would get into the flow of 
writing our lesson and then we would stop and do the big conceptual story, but it wasn’t really necessarily the 
lesson we were working on.” Another teacher noted challenges with balancing the use of the videos they were 
working to include in their lessons, with finding that the videos did not fit their new understanding of what 
should happen in the Explore phase of the 5E learning cycle. Despite these challenges, Treatment 2 focus 
group participants generally agreed that Week 2 helped to build their understanding of concepts introduced in 
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Week 1. “Whenever I started trying to plan lessons along with the 5Es on the lesson plan sheet, it was very 
slow and very tedious and I didn’t quite understand exactly how long it was. [By] the end... it was flowing 
and I could do it faster with practice, [after] during the week doing it each day.” Several teachers conveyed 
that larger chunks of uninterrupted time would help: “Analyzing their lesson plans was like [reference to 
other teacher] said, at first, it was absolutely needed. ... Then it got to the point that we did know what we 
were doing and it was flowing, but we were being stopped.” All Treatment 2 focus group participants 
reported feeling well prepared to take their learning back to the classroom and integrate what they learned 
during Week 1. 


Treatment 2 teachers made the following comments in response to the focus group prompt, E/ements 


of this professional development experience from which I learned most: 


For me it was the 5Es and how they work together to take the topic... to that deeper level. I think I alvays 
stopped and explained and never really went down to the bottom of how important that bottom part was... . 
It was important for me to see that and work through it myself as the student to understand how much deeper 
it took my thinking to carry over to the next topic. Week 1 being the learner, that’s what helped me a lot as 
well. 


Mine is... the content that we learned. 


I think with me I can maybe be too proactive with things and maybe show [students] too much so the idea of 
when I go back and letting them explore and problem solve as a team rather than step by step... because 
they can problem solve like this the first week. I think that will pose the biggest challenge for me, because it’s 
a gray area and we need to provide them enough to where they can at least get started. ... My challenge is 
going to be where does that stop? 


I would agree with [other team member], just to go back to that. I just ahvays feel I need to give my students 
all this information and even though I might rush into that first... . I think that would be hard seeing them 
Jrustrated, but sticking to it like the class leader did would be really good. 


Just the idea of the storyline to have the linking with the Rey ideas. 


When asked about what elements of the professional development could be improved, Treatment 2 


teachers offered the following comments: 


In the afternoons in the first week. where we were just basically sitting there worrying but we still wanted to get 
up and do [the activities] at the same time. Maybe we could still do those same activities, but instead of 
moving into another room... we could just get up and do the activity but come back to the same spot where 
we were at. I think that would be beneficial... . I think that the first day, do the first two units and do the 
storyline and do the planning, and then the next day you get the whole day to lesson plan and do those two 
lessons and share it with people. That way you have the entire day to dig into it and some people even got up 
to be able to practice. 


I think for me, it’s just the lesson feel, not disrupting the flow of the lesson plan, if there was some way to 
definitely have a plan and we were going to do the conceptual story, do lesson planning, instead of the 
choppiness. Especially when you are really trying to mull it over with a team, for consensus, and then you get 
disrupted, it’s really hard to get back to the consensus. My biggest issue was just the constant sitting. I heard 
that from a lot of people, that we're always sitting down. We're pretty active, so finding a way to get a little 
pep in our step and better utilizing these chairs. 


I think I'd like normal lectures, maybe that first week in the morning. It’s necessary that it has to be done, 
but I think by the afternoon after you've gone through all of that deep thinking, it felt like, “Ah.” 


Front-load it with the storylines and give more time than an hour to do some planning, because it seems like I 
was just starting to get to something and we moved onto something else. 
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Preservice Teachers 


Objectives. Preservice teachers differed in the depth of their understanding of the QuEST project’s 
objectives prior to the start of Week 1. Some preservice teachers had general expectations of learning more 
about science at the elementary level and anticipated a dual focus on learning material and learning to 
implement what they learned. Other preservice teachers shared more specific expectations, anticipating 
properties of matter as the content area, and “the overall goal... by the end of the week, you should be able 
to take all six learning cycles, the 5E, the conceptual storyline, UDL, and then be able to implement that with 
your group of students.” Another preservice teacher conveyed more implicit objectives: “To challenge our 
misconceptions about matter, being more investigative with our teaching, not just telling them the answer. 
We were learning about ourselves as teachers.” Once Week 1 began, preservice teachers who were less clear 
about the project objectives conveyed they had a better understanding of the project goals for the first week 
and what to expect leading up to Week 2. 


Materials. Preservice teachers were surprised and pleased with the quality and variety of the 
materials they received as part of participating in the summer institute to take back to their schools: “T like 
how we have a purple balance, not as nice as the red balance, also these electric balances, [so] then in the 
classroom ... you can talk more about how science is measuring and about accuracy ... so going in to how 
we get accurate readings besides just getting the right reading.” Another preservice teacher noted the 
reasonable cost of the materials, which would allow them to conduct the same experiments in their 
classrooms. 


Delivery and Organization. All preservice teachers appreciated how well organized and prepared 
the facilitators were; as one preservice teacher stated, “They were very well prepared, which helped the 
organization as a whole. That was easy to tell.” Regarding organization, one preservice teacher said, “I think it 
was put together really well. I was never lost or trying to figure out where to go or what to do next.” Some 
preservice teachers conveyed feeling a little frustrated during the first week, feeling behind compared to their 
group members or wanting validation or feedback. However, preservice teachers recognized the value of 
being placed in a learner’s role as helping them to get a sense for how their students would feel in a similar 
situation. 


Facilitators. The preservice teachers reported a sense of cohesion and consistency of responses 
among facilitators and staff members. “I’d say they all worked really well together. . .. With that big of a staff, 
I feel like a lot of times you might get two different answers from two different people, but it seemed like if 
you asked four of them the same question, you were going to get the same answer every single time.” Other 
preservice teachers mentioned the quality of interactions with the facilitators that helped build their 
understanding: “I felt they asked good questions that pushed us towards the goal of the lesson” and “They 
gave us lots of different ways to look at things.” 


Applicability/Preparation. Preservice teachers found the 5E model to be valuable, saying, “You 
don’t have to be that specific with it... . It’s very adaptable to your students.” Another preservice teacher 
shared in reference to her Week 2 students asking many questions, “I like how flexible it is... . We spent the 
entire three hours explaining, just explaining ... that was really great because we were going at their pace.” 
The preservice teachers shared mixed opinions about UDL. Two preservice teachers noted the many pieces 
that go into the implementation of UDL and the anticipated challenges to execute UDL effectively to meet 
the full range of different students’ needs. Another preservice teacher reported finding the UDL strategies 
helpful, particularly in IEP meetings with parents: “UDL is just something any parent, or any teacher with a 
student with special needs, would like to hear. It’s just saying they have a way to get involved specific to them 
[the student].” 


Professional Development Experience. All the preservice teachers valued the opportunity to teach 
students during the second week, with support from current teachers. “Well, just making the lesson plan and 
applying it to an actual class, that was something that I personally haven’t done before.” Another preservice 
teacher shared, “That helped me not to panic, when they don’t get it the first time you deliver it. I was like, 
‘Oh, they don’t understand, what do we do now?’ Now I know, okay relax, just try to do it another way.” 
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Several preservice teachers suggested they would benefit from collaborating with people from other groups to 
get new ideas. The preservice teachers shared several key ideas they took away from the second week, 
including the importance of behavior management, the need to differentiate instruction to engage advanced 
learners, the need to “find ways to motivate” students or keep them from becoming distracted, the broader 
recognition of the different ways that students think, and the need to adapt what they do to meet the needs of 
each student. 


Preservice teachers made the following comments in response to the focus group prompt, Elements of 
this professional development experience from which I learned most: 


Adapting lessons for all and investigation activities. 
Using the UDL model. 


I feel like actual classroom application of the lesson plans. As students, we write a lot of lesson plans and we 
don’t necessarily get to apply them. 


I'm just always remembering that as a teacher, you're still a learner. I feel like a lot of the time teachers just 
think they’re the authority and that the students are going to learn from them. But putting us in the learner 
role, I learned that even after fourteen years of school, I learned stuff that I’ve never even know before. I feel 
like we're students all the time. 


I liked how instead of just telling kids what to say, you're going to probe each of them to tell you and use what 
they say to answer their own questions. I was thinking that I never really thought of that before, so I’ve really 
been trying to apply that this week, which is awesome. 


Just how professional and organized [the QUEST Institute] is... . That’s something that you can use in 
your own classroom because they alvays know what’s going on and then also the application. [The application 
of the lessons is] something that’s very useful, because you can never have too much field practice. 


When asked about what elements of the professional development could be improved, the preservice 
teachers offered the following comments: 


Even more collaboration between the teachers. 
I think more lesson time, lesson planning time for the camp. 


Just for my own perspective, I Rnow that I knew a lot about 5E and UDL, so I feel like having a poll or 
something of the people who had [previous knowledge] and grouping accordingly. Then you can put one person 
you know has had to teach then they can help explain and maybe engage more deeply with the content. 


I'd probably say when we had to the first week with the check-in of your instructor... kind of varying, 
maybe even asking a specific person each time. I had someone in my group who answered every single time and 
the rest of us may not have been to that point... . It wasn’t really testing everyone, it was just one spokesman 
for your group and then they just let you go. 


One thing that was kind of hard for me was, I liked learning the week before but having to use those same 
key ideas in that order. I know we can do our own thing but that was hard for me, especially lesson 2B, our 
group really struggled pulling things from there and applying it to kids and that kind of way. You know, 
more applicable ways to apply it. 


I'd say they hit everything. I'm trying to think. I like the more collaboration on Week 2, because I feel like 
there are a lot of ideas out and my group was kind of going back to the same thing over and over. 
Project Staff Interviews 


During project staff debriefs, evaluators asked staff members to reflect on what went well, what 
challenges they experienced, and recommendations to improve the functioning of future institutes. To help 
frame their thoughts, staff members were asked to consider pre-summer institute activities, Week 1, and 
Week 2 separately. 
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Across all staff groups, the following themes were identified in terms of what went well with the 
institute: 


e Highly engaged and motivated teacher participants, with less interpersonal conflict than the previous 
year 

e Increased confidence and knowledge of team members 

e =Plexibility in response to participant needs 

e Better preparation with technology needs for teacher participants 

e Continued high levels of organization and anticipation of teacher needs 


e Use of the pre-summer institute research institute for the QUEST staff to establish roles and 
responsibilities, as well as to pilot new curriculum materials 


Themes that were identified as challenges across all participant groups included the following: 


e Recruitment and teacher/school attrition 
e Smaller than expected meeting space for Week 1 activities 
e Loss of project manager in the spring 


In addition to the common themes identified above, there were positives aspects and challenges 
unique to each participant group, as summarized below. 


Principal Investigators. The principal investigators were in agreement that the 2016 QuEST 
summer institute was successful, noting the continuation of excellent organization of all aspects of the PD 
experience and challenges that were limited and relatively easy to overcome when they did arise. Team 
members agreed that the overall tone of the summer institute was positive. Similar to last year, the staff 
research retreat was identified as an instrumental piece of the coordination process, allowing all team 
members to mentally prepare for the two-week institute. The principal investigators reported that the staff 
said they were less stressed, having more time to get on the same page in terms of roles and responsibilities 
and to prep materials for the institute. The teacher participants contributed to the overall preparation with 
timely submission of materials, allowing the pre-institute data collection activities to proceed smoothly. The 
staff also noted that having a postdoctoral researcher on board throughout the school year and leading up to 
the institute was an asset as well, as he was able to contribute in multiple ways, especially being available to 
conduct observations and provide support to participating schools. The main pre-institute challenge reported 
was the resignation of the project manager in the spring. Although the QuEST staff needed to work through 
some snags related to project management, there were no major issues and they were able to successfully 
hand over institute tasks to a part-time helper. During the interim, several project staff members worked to 
fill in where needed, attesting to the flexibility of the project staff in general. 


Principal investigators reported that during the summer institute the teacher participants worked well 
with each other, and there was less interpersonal conflict than in the previous year. The PIs were happy with 
the way the final curriculum turned out. Although the curriculum was not fully developed along the planned 
initial timeline, the team was able to pilot the activities more extensively this year at the research retreat and 
during alumni activities, which the staff considered to be helpful. During Week 1, an additional science 
education graduate student with content area expertise was brought in to provide support during the content 
learning sessions, which the staff found to be very helpful. Also noted as helpful to the facilitation of Week 1 
activities was a change in the method used for lesson checkouts. The changed method decreased the feeling 
within groups that they were ahead of or behind their fellow participants, so there was less tension in the 
groups to hurry without paying close attention to the materials. Afternoon sessions also went well. There 
were fewer problems with technology this year. As for the content of the afternoon sessions, the principal 
investigators perceived that the teachers got into the material easily and quickly. It also seemed to the 
presenters that the materials around the conceptual storyline were better received and understood this year, 
likely because the presenters had more time to develop the presentations across the year and to make better 
connections to the assessment component guided by the postdoctoral fellow. The team noted that the set-up 
and download of data from the teacher assessment was smooth, allowing the graduate research assistants on 
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the research team to function efficiently. A challenge noted in Week 1 was that smaller working groups of 
teacher participants meant more checkouts for the staff to conduct, but this was balanced out by having more 
interaction in the groups. The smaller size of the room used this year meant that participants were somewhat 
crowded together, increasing the noise level and making moving freely a mild challenge. The topic this year 
also was seen as a mild challenge, as properties of matter was an unfamiliar topic for most teachers. 


Week 2 summer institute activities were viewed as successful. Kids QUEST was full again this year, 
and the principal investigators reported that parents began emailing about their child’s participation in the 
summer program as early as last winter. Teachers who worked with students this year seemed more confident 
with the idea of teaching in Week 2, and teachers reported fewer behavior issues than Treatment 1 teachers 
had the prior year. Revised reflection forms were helpful to the staff, allowing them to better target support 
needs. The assigned support team method for providing assistance to Treatment 1 teams also worked well, 
along with a revised process for supply retrieval and return. One principal investigator commented that 
teachers this year used their own materials and ideas to a greater degree than groups had in the past. 
Challenges noted in Week 2 were attendance issues and a last-minute reassignment of students to different 
rooms on the first day. 


I think we worked out a lot of kinks. I think there will ahvays be some that come up, but it’s made me 
realize even more, I’m amazed that we do it with as few people as we have, that it could be a much, much 
larger team doing this... . All the things I was worried about prior to the institute didn’t come to fruition, or 
were non-issues.... They [the things worried about prior to the institute] a// got integrated into the 
procedures we already had, so that was nice to see because we've had a new project manager for every summer, 
and still it got better and better despite that. I think that says a lot about the core group that works, that 
despite all the personnel changes and additions, last-minute things, it all went really smooth. 


I think overall it was actually really successful. I think we met our goals. I think we did an excellent job with 
delivery .. . content, materials, making all the different pieces fit together and work together... . We could 
always look at what we're doing and improve upon it, but I would say as far as the structures and the systems 
and the supports and doing what we're doing with what we have, I think it was very successful. 


I'm really looking forward to this year and digging into the research because we're not writing a new 
curriculum, we're not recruiting a new cohort of teachers, we are not developing new assessment instruments, 
we're not recruiting a whole other set of kids, we’re not making room reservations, ordering parking passes, 
catering meals. None of that has to happen now... . That really consumes a lot of time, even when someone 
else is doing it, to just delegate and oversee. I think that’s going to be really, really good for us. 


Faculty/Teacher Instructional Staff. The QuEST teaching staff reported that the summer 
institute ran even better this year than in past years, building on the program’s history of strong organization 
and communication. The staff research retreat was again recognized as a critical tool in preparing team 
members for the professional development activities of the summer institute, helping to solidify the 
culmination of the past years’ activities. The staff felt better able to fully integrate the different professional 
development strands, including such activities as sharing presentation materials, piloting and practicing newly 
developed activities, and reviewing a new evaluation tool in a face-to-face setting. One staff member reported 
that a QuEST teacher alumnus who participated in the ReQuEST session felt that being involved in the pilot 
activity was a valuable experience that would in turn allow them to use the activity in their own classroom. 
Although all staff members noted that they had been slightly worried about having a new project manager 
going into the busy summer season, they reported that the replacement person fit into the team very well and 
was able to fill the role without any difficulty. 


During the summer institute, the teaching staff noted a positive group dynamic with high levels of 
excitement and engagement. Check-in activities for the teacher participants were more efficient this year. 
Examples of methods used to increase efficiency included having several staff members on hand to ensure 
that teacher dormitory check-in proceeded smoothly, meeting people at the dorms to help with orientation to 
the campus, and having staff members strategically placed in the mornings to help with parking passes and 
directions. The Week 1 sessions seemed well received by teachers, and all presenters reported having more 
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confidence with the materials and being able to anticipate and respond to questions better. One staff member 
mentioned that the new reflection sheets helped to facilitate better and more targeted participant feedback. 
Technology issues from last year did not arise because things got set up earlier in the week. Some staff 
members reported feeling some concern before the summer institute that the curriculum would be too easy, 
but it turned out that there were a lot of misconceptions to correct and new understandings to build among 
the teacher participants. The staff also believed that it was helpful to have another science content specialist 
to assist with questions during Week 1. Challenges noted during Week 1 included room space and some 
challenges with the materials included in the teacher participants’ kits. This challenge, however, was easily 
sorted through and the staff reported it actually led to a rich scientific discussion around precision and 
accuracy. 


The staff reported having a better understanding of the goals for the outcome of the Treatment 2 
group in Week 2 this year. Some teachers still struggled with the conceptual storyline content at first, but a 
new activity this year that included the narration of a conceptual storyline poster between teacher and 
students seemed to help facilitate understanding, allowing teachers to understand the narrative flow. The staff 
believed that teachers dove into the UDL chart early on and deeper than in previous years. Additionally, 
Treatment 2 participants were reported as happy to have the work time they received immediately after Week 
1 PD activities, and there were no comments that teachers would rather be participating in the situated PD 
experience. Challenges noted by the staff members working with Treatment 2 teachers were focused on 
curriculum constraints expressed by the teachers, who were concerned about how they would be able to 
implement what they learned in their district/school. These staff members thought this concern was 
alleviated for teachers as the week went on and they came to a better understanding of how they could 
structure their current curriculum using their new knowledge and understanding. One staff member provided 
an example of a group from a school with a scripted curriculum. At first, they were very concerned about 
how they would be able to integrate the PD materials into an already prepared curriculum. But as the week 
went on and they analyzed the lessons for the conceptual storyline, they came to their own understanding of 
where they could add lessons and drop content to make the storyline stronger. The teaching staff working 
with the Treatment 1 teams noticed that some teachers struggled at first in terms of planning and moving 
through the activities and letting students work independently; however, the feedback sessions at the end of 
each day seemed to allow them to gather and share ideas and work through their challenges. The staff 
commented that teachers were better able to spot areas where they could add in brief assessments of 
understanding by the end of the week. 


Graduate Research Assistants. The GRAs agreed that the research retreat has been a helpful 
feature of the pre-summer institute activities during the past two years. Specifically, the retreat is viewed as a 
means to bring everyone together and use the time to wrap their heads around what is coming in the summer 
institute, to reestablish roles and responsibilities, and to ensure that everyone is familiar with where the 
different expertise lies on the team. This year, the GRAs reported that time was used during the retreat to 
pilot a new curriculum activity. This was helpful because it allowed the team to recognize potential hurdles 
that participants might have with the activity and related content and thus allowed them to work through 
solutions ahead of time. Piloting the activity was also seen as helpful to members of the team who are not 
content experts, giving them a sense of the curriculum content and activities ahead of time. Additionally, the 
GRAs reported that the teacher data collection activities were completed prior to the beginning of the 
summer institute, a marked improvement over the prior year. The team attributed this success to a 
combination of being even more organized and experienced this year, and to this group of teacher 
participants being especially well prepared. One difference this year that was noted by team members who did 
not have a science background was that this year’s topic was less familiar to them, so finding ways to make 
connections between the content and UDL was mote challenging. 


The GRAs perceived that Week 1 activities proceeded very smoothly. The summer institute began 
with a more successful move-in for teacher participants staying on campus. One GRA reported that having 
Dr. Thomas on hand to help with the move-in activities was particularly helpful. Additionally, a phone app 
was introduced to teachers as a tool to help them navigate their way around campus. One GRA noted three 
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changes this year that helped to better facilitate the flow of the morning content lessons. First, the groups 
were purposefully set up differently this year, so that teachers from the same schools were split up across the 
tables. Another change to the process was that instead of writing where groups were in their progression 
through the lesson with their checkouts up on the board, the table numbers were written on the board. This 
seemed to allow teachers to focus less on feeling like they had to be at a certain point but to continue at their 
own pace. Finally, the lesson materials were handed out to teachers when they needed them, instead of having 
everything in the binders ahead of time, allowing teachers to stay focused on the task at hand. In the 
afternoon sessions, GRA presenters reported that the opportunities across the year to practice presentations 
around conceptual storylines and UDL increased their confidence with the material, which they believed 
resulted in increased teacher understanding of the materials. One challenge identified during Week 1 was the 
lack of space in the room where the group met each day. Due to space restrictions, the QuEST staff not 
participating in facilitating a session did not sit at tables with teachers. There was some disagreement among 
the group as to whether or not it was better this year with no QuEST staff member sitting at the tables with 
the teachers during the instructional sessions. Some GRAs believed that it was better to not have QUEST 
staff members at the tables to inadvertently prompt or direct teachers. Others thought that by not sitting with 
the teachers, they were missing out on the places where teachers were getting hung up in the curriculum and 
were thus less likely to be able to support them to the extent that they may have been able to. Another 
challenge reported was minor equipment issues with the material kits, but the GRAs believed this challenge 
was easily remedied. Another change that was reported to be helpful to the group was the addition of a new 
student team member with a science content specialty to assist in the instruction during Week 1. The new 
GRA team member said that although it took some time to determine individuals’ roles initially, he felt 
comfortable with his own role on the team early on. The student said that he noticed immediately how well 
prepared the entire team was. He noted that even when they did not have time to give him a full introduction 
to an activity, he was able to fit in and play off their moves and find the pattern that he could fit into. 


The GRAs who worked with Treatment 2 teachers reported that the Week 2 activities were very 
organized yet intentionally flexible, that the plan allowed them to navigate outside the plan when needed. 
GRA team members reported that the structure and content of the PD worked really well this year, noting 
that teachers were able to dig in and develop plans around their own classroom/school needs. Additionally, 
one team member believed that the inclusion of special education teachers in Treatment 2 helped in 
developing deeper discussions about the use of UDL, reinforcing the importance of including UDL in 
classroom planning. In general, Treatment 2 GRAs reported that the teachers appreciated the planning time 
and support they received. GRAs working with Treatment 1 teachers and preservice teachers commented that 
teachers seemed confident with the materials and did a good job transforming lessons from Week 1 into kid- 
friendly activities. Additionally, Treatment 1 teachers seemed to have a good handle on instructional and 
behavior management techniques and made good use of their team advisors by running ideas past them 
during the reflective sessions. Treatment 1 GRAs believed that the system of having just two of the 
Treatment 1 staff members assigned to two groups worked well and was more efficient overall. GRAs who 
also worked on the data collection and analysis activities that occurred during the summer institute reported 
that these proceeded smoothly and in a very organized manner. Overall, QUEST GRAs reported very few 
challenges during Week 2. Those that were reported were mainly minor issues with the lesson materials. It 
was noted that, during Week 2, one Treatment 1 team had a teacher who wanted to explore an extension to 
the content although not everyone in the group was comfortable with this trajectory. The situation was felt to 
be effectively resolved through the use of strategic feedback and coaching. 


Everybody is always... five steps abead. I was just so impressed in general with the way the Rits were set up, 
the way they had everything organized, living situations, and just... everything you could possibly think of, 
they had already been prepared for us. It was impressive. 


For me I was the most confident, this being the third year, and because we had done so many presentations on 
my topic area. This ts the first year I felt like I could open them up, not that I ever held back an opinion in 
the past, but really content and design-wise I was able to really speak up and make my own decisions then 
defend my decisions. I was more of a contributing member than I had been in the past. That was true on my 
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UDL team, but that was also true when I'm working in other disciplines... . I got the most out of that this 
year, just working between departments and seeing how other people think and knowing “oh that’s your 
process”... but we can meet on this ground. 


Mine [takeaway lesson from QuEST] is always related to data collection. I learned a lot as a GRA. I 
think... this being the third year, [I’m continuing to learn] what works, and what needs to be collected, and 
the data needed to give you the information you might want or to answer particular questions. 


For me, I feel like I'm always refining how to teach and learning how others learn. QuEST is such a 
wonderful place for me to do it, not only because there are so many teachers there, but also talking through 
certain ways they're coming to me with a certain perspective. And then there’s also the full QUEST team that 
have all the expertise. I always feel like I’ve learned quite a bit with every QUEST workshop. I’m able to 
refine my teaching technique, and all the tools that I have in order to be effective in the classroom. 


Academic Year Saturday Sessions 2016-17 


During academic year 2016-17, quarterly full-day Saturday sessions were held to provide QUEST 
teachers a chance to regroup and support one another in applying what they had learned in the summer 
workshops. The structure of the sessions included facilitation by the QuEST project staff on a predetermined 
topic, integrated with teacher share-outs around the topic of the day and how they are using QuEST-related 
practices in the classroom, as well as opportunities for problem-solving with colleagues. 


An EPIC evaluator attended one of the four Saturday follow-up sessions, on April 8, 2017. A 
professional development observation tool was used to determine the presence of best practices for 
professional development (see Appendix). The same tool was used in the summer institute observations. 


Results of the Saturday session follow-up observation are summarized in Table 3. Most of the 
standards were observed during the Saturday session and those that were not were standards that either did 
not apply or would have been challenging to observe during this type of one-day session. 


Table 3. Summary of Professional Development Standards Observed in Spring Follow-Up Session 


Standard/Indicator April 2017 Session 


Selection and Organization of Content 


Clearly stated the purpose of the instruction 


Defined the target audience and necessary prerequisite skills 


Solicited and answered questions about their instructional goals/activities from teacher 
participants where necessary 


Clearly defined expected outcomes 


Formulated a limited number of goals and/or objectives 


Developed a conceptual framework to highlight major ideas to be presented and to 
organize the content 


Research 


Described the research, evidence, theory, or reports from practitioners that supported the 
content 

Avoided conjecture and lots of personal opinion, but included valid alternate opinions to 
help the participants make their own judgments 


Fully cited all references and research 


Identified professional standards or content standards where applicable (e.g., CCSS, @ 
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Standard//ndicator April 2017 Session 


NGSS, GLES) 


When possible, cited evidence of effectiveness of practices presented in the training or 
presentation 


Principles of Adult Learning 


Drew upon and honored learners’ prior knowledge and experience 


Provided opportunities for learners to connect new learning to their own work 


Designed and used problem-centered activities 


Provided strategies, tools, and techniques that participants could begin to use 
immediately 


Offered opportunities for participants to assume responsibility for their own learning (i.e., 
to be as self-directed as appropriate given content) 


Community of Learners 


Encouraged dialogue and sharing among participants 


Encouraged dialogue and sharing between participants and PD staff/content experts 


Incorporated opportunities for team building and collaboration during sessions and in 
designed follow-up 


Considered structures and practices that connect participants across time 


Encouraged risk taking 


Provided for celebration of small successes and learning from failures 


Practice 


Incorporated opportunities for participants to practice new skills in a safe workshop 
environment and to receive feedback from facilitator and colleagues 


daily instruction, including content, process, and addressing student misconceptions 


Facilitated student teaching component in a clear and organized manner N/A 
Encouraged the use of innovative and/or novel approaches where appropriate @ 
Ensured adequate time for lesson planning and team building before each teaching N/A 
session 

Provided guidelines for analyzing student work and using data to guide future instruction @ 
in line with PD goals and activities 

Included debrief time with content experts and PD facilitators to provide feedback around N/A 


Designed follow-up activities to support teachers in new content and process 


Included structures to track participants following the professional development 


Encouraged peer classroom observations when appropriate 


Developed capacity of individuals at site to provide leadership for follow-through 


Diversity 
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Standard//ndicator April 2017 Session 


Established norms of respect, openness, and listening through both the content and e 
process 

Used language and media elements that were respectful of and appropriate to all oO 
Planned for an inclusive approach (i.e., used strategies to engage all individuals and @ 
subgroups) 


Selected resources that brought diverse points of view to the table 


Provided opportunities for educators to share knowledge, skills, and strategies for 
involving families or other stakeholders appropriately 


Process Design 


Incorporated a variety of presentation strategies and activity formats 


Included media elements that appropriately support the content 


Ensured that participants have opportunities to develop or review related knowledge 
base 


Varied learning activities, interspersing didactic with active, hands-on learning 


Structured daily activities and kept on task so that adequate time was available for 
instruction, activities, and reflection 


Self-Assessment 


Ensured that participants were clear about expectations for their own learning and 
change 


Allowed for practice and assessment of all stated outcomes 


Encouraged participants to identify and build on their individual strengths 


Provided opportunities for participants to identify barriers to change 


Evaluation 


Built in debriefs at critical points in the training (e.g., at the end of a day) 


Sought participant feedback throughout the formal learning experience — both formally 
(e.g., quick feedbacks at day’s end) and informally (e.g., by asking participants to signal 
how it’s going for them at intervals throughout the learning experience) 


Scheduled ample time for end-of-session evaluation 


Teacher Survey Spring 2017 


In the spring of 2017, EPIC administered a survey to teachers to gather information about their 
experience with the QuEST project, specifically the impact of the professional development received during 
the 2016 summer institute and the follow-up sessions that occurred during the academic year. The survey 
consisted of 30 items. Additionally, five demographic questions were included on the survey. Each survey 
item was mapped to the overall evaluation framework, ensuring adequate data collection for each project 
objective, including science curriculum descriptions for participating schools, implementation of QUEST 
project curriculum, development of understanding of science content knowledge, Universal Design for 
Learning, the 5E model of instruction, and attitudes about the quality of the professional development 
received. The survey was administered using an online format and designed to be completed in approximately 
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20 minutes. Most questions were closed-ended, using a 4-point Likert scale (Strongly Disagree to Strongly Agree). 
Eleven open-ended items were also included. Survey questions are included in the Appendix. 


Teachers were invited to complete the survey during the spring Saturday follow-up session. All 
teachers who attended the 2016 QuEST summer institute were invited to participate. Participation was 
voluntary and respondents were assured that their responses would remain confidential, to be used only for 
research purposes. Teachers who did not attend the April 8 follow-up session were emailed the survey link 
and given four weeks to complete the survey; teachers who had not completed the survey were prompted at 
the beginning of the third and fourth weeks of the administration window. There was an overall response rate 
of 73% (32 respondents out of a total of 44 possible). Open-ended data were analyzed and summarized 
thematically using NVivo qualitative research software. Closed-ended items were analyzed using descriptive 
statistics. 


Prior to completing questions related to the summer institute, teachers were asked to provide 
information on their curriculum and instructional practices. Three respondents (16.7%) reported being a 
preservice teacher during the 2016 summer institute. Among the practicing teachers, there was some 
variability in the type of curriculum used, with the largest percentage of teachers selecting the Osher category, 
followed closely by the percentage of teachers reporting use of a district-created curriculum. In addition, most 
teachers reported engaging in science instruction several times per week (35.5%) or daily (64.5%), with no 
teachers reporting engaging in science instruction once weekly or less often. More than half of respondents 
(54.8%) indicated the typical science lesson was 31—45 minutes. Further, most responding teachers noted they 
taught their students about the properties of matter during academic year 2016-2017, with many teachers 
(32.3%) covering the properties of matter for several learning cycles; more than half of respondents taught 
the full unit. Less than half of responding teachers (41.9%) reported that the properties of matter instruction 
was very close to the summer institute. Respondents also reported spending an average of 1,144.8 minutes 
(about 19 hours) teaching students about the properties of matter. Table 4 summarizes this information. 


Teachers were asked to indicate how qualified they felt to teach science, physical science, and the 
properties of matter in the spring of 2016 when they completed the enrollment survey for the program and 
then again on the spring 2017 survey, after their participation in the summer institute. As shown in Figure 1, 
most respondents noted feeling adequately qualified to teach science in general (84.2%) before their 
participation in QuEST, with a small percentage of teachers (2.6%) reporting feeling not well qualified. After 
participating in QuEST, all teachers reported feeling adequately qualified (59.4%) or well qualified (40.6%). In 
regard to teaching physical science, on the previous survey most teachers (76.3%) reported feeling adequately 
qualified to teach physical science and 7.9% felt we// qualified to teach physical science, with 15.8% of teachers 
reporting feeling not well qualified to teach physical science. In the spring of 2016, most teachers reported 
feeling well qualified to teach the properties of matter, with 62.5% of teachers selecting this response. On the 
enrollment survey, most teachers (65.8%) reported feeling adequately qualified to teach the properties of matter, 
with 23.7% of respondents selecting not well qualified and 10.5% selecting well qualified. This item showed the 
greatest growth in self-perceived qualification from summer institute enrollment to nine months post- 
institute. Wilcoxon Signed-Ranks tests were run for each of the pre- and post-survey comparisons shown in 
Figure 1. The output indicated that post-test scores were statistically significantly higher than pre-test scores 
for each comparison (Teaching general science x = -2.65, p = 0.008; Teaching physical science x = -2.33, p = 
.02; Teaching properties of matter z = -4.07, p = .001). 


Next, teachers were asked to report how their instruction has been influenced by their experiences as 
a learner of science in the summer institute. Of the 32 respondents, 31 provided an answer to this question. 
Many teachers reported the experience of being a learner at the summer institute allowed them to better 
consider the student learning perspective in their science instruction and to use questioning techniques to 
encourage students to engage with the content more deeply. Teachers also reported that they gained an 
increased understanding of the 5E learning cycle method and how to implement it in their classroom. Other 
respondents noted that they gained content knowledge and that their experience in the summer institute had 
shifted their thinking about the nature of science instruction in general. See Table 5 for a sample of the 
comments provided by teachers. 
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Table 4. Teacher Information on Science Curriculum and Teaching Practices 


properties of matter? 


Teacher Information n % 
Were you a preservice teacher during the 2016 Summer Institute? 
Yes 3 9.4% 
No 29 90.6% 
Do you use a published curriculum or is the science instruction teacher-developed? 
Commercially offered curriculum from a publisher 4 12.5% 
District-created curriculum 11 34.4% 
Teacher-created materials 5 15.6% 
Other (please specify) 12 37.5% 
e We have Discovery Techbook plus district and personal are allowed 
e = MySci by Wash, district supplements 
e We use district and teacher created curriculum 
e We have a textbook, but we have piece mealed [sic] activities from the district 
e = This year we used Mystery Science 
e _|used information from resources | developed and textbooks 
How often does science instruction happen in your classroom? 
Daily 20 64.5% 
Several times per week 11 35.5% 
Once weekly 0 0% 
Less than once weekly 0 0% 
What is the typical lesson duration of the science instruction in your classroom? 
30 minutes or less 4 12.9% 
31-45 minutes 17 54.8% 
More than 45 minutes 10 32.3% 
In the 2016-17 school year, did you teach your students about the properties of matter? 
Yes; a full unit 17 54.8% 
Yes; several learning cycles 10 32.3% 
Yes; only one learning cycle 2 6.5% 
No 2 6.5% 
How close was the properties of matter instruction to the unit that you experienced in the summer institute? 
Very close 13 41.9% 
Close 15 48.4% 
Not very close 3 9.7% 
Instructional Time Spent on Properties of Matter M SD 
How much total instructional time (in minutes) was spent teaching students about the 1,144.8 | 1,212.0 
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2017 Pre-Institute 


How qualified do you feel n=38 


to teach science, in 


general? 
2017 Post-Institute 


n=32 


2017 Pre-Institute 


How qualified do you 
: i n=38 


feel to teach physical 
science topics? 
2017 Post-Institute 
n=31 


2017 Pre-Institute 


How qualified do you feel nea 


to teach properties of 


matter? 
2017 Post-Institute 


n=32 


QuEST INCREASES PERCEIVED QUALIFICATION TO TEACH SCIENCE 


Well Qualified Not Well Qualified 


aE 
eae 


Adequately Qualified 


84.2% 


59.4% 


76.3% 15.80% 


T4.2% 


~ 65.8% 


23.70% 


37.5% 


Figure 1. Teacher Perceptions of Their Qualification to Teach Science-Related Topics. 


Table 5. Teacher Perceptions of How Instruction Has Changed as a Result of QuEST Participation 


Theme n Representative Comments 
Student self- 13 e From the summer institute, | feel like | am better at questioning my students in the 
discovery classroom to get them to think on a critical level. Students are continuing to 
question how they think about science. 

e It has helped me to remember what it’s like to be a student and confused on what 
to do. It also helped to realize what problems my kids may face. 

e By being a learner during the summer, | was able to put myself in the shoes of my 
student. Asking the questions they would ask, then finding the answer by discovery. 
This helped me to better understand the content which in turn helps me instruct 
better. 

Increased 11 e A lot of commercially written lessons just touch the surface of the 5Es. The Summer 
confidence in Institute gave me the opportunity to more fully understand the possibilities of the 
and use of 5E 5E structure. 

e | have been able to implement the 5E Learning Cycle into my teaching throughout 
subject areas. It has been a more accurate representation and measure of student 
learning. 

e | learned that the use of the 5E model is more effective than the traditional 
teaching | was used to before participating in QUEST. 

Changes in 9 e | gained a lot of knowledge and feel more comfortable with my curriculum. 

content e | feel much more knowledgeable about the materials and subject matter. 

knowledge e | have a better knowledge base for the properties of matter, which makes me 
confident in teaching that subject matter. 


Educational Policy Improvement Center 


28 


Evaluation of the QUEST DRK-12 Grant: Annual Report for Year 4 


Theme n Representative Comments 
Reflection on 7 It has made me think deeply about simple science concepts such as what is 
the nature of happening to particles. It has also made me use more vocabulary to help students 
instruction in understand and solidify the concepts taught. | was able to look at science through a 
science different lens of how | even approach science instruction as it pertains to students 
at different levels of learning. 
I’ve got a lot of ideas that replaced old ideas on how | teach and manage science. 
Increased 5 Extremely! | was not confident in physical science as an educator. | was comfortable 
confidence in with my science units in 4th grade. As | transitioned to 5th grade | found myself 
science feeling like a first-year teacher again. The QUEST program allowed me to question 
instruction my students in the right direction, rather than questioning them as if | had no idea 
where | wanted them to go or end up. 
| enjoyed having the lesson plans available to use during my instruction of the 
matter unit. | could use, tweak and extend many of the lessons. But having the 
experience and lessons, better equipped and increased my confidence when 
teaching. 
Increased 3 It has been very much influenced, it has allowed me to look at science a lot 
awareness and differently and teach it very differently using the UDL model. 
use of UDL It has enlightened me on multiple levels. | am better able to understand balancing 
UDL and the 5 E’s in my science lesson and walking away with multiple teaching 
strategies and tools. To differentiate instructions in the special education 
environment. 
Understanding 3 Considering the 5E lesson model and using a conceptual storyline is greatly in the 


use of 
conceptual 
storyline 


forefront of my mind as a result of the Summer Institute. 

From the Summer Institute, | feel that my instruction is now heavily influenced from 
my experience as a science learner. | now see how important it was for me to tap 
into my own background knowledge before diving into the more in-depth 
information. This is something | have been doing and intend to do in all future 
lessons. | am also checking my instruction to ensure that | have conceptual 
storylines and use the 5E model a majority of the time when planning science 
instruction, as this helped me learn. 


Another portion of the survey focused on teachers’ confidence in their ability to meet the needs of 
diverse learners before and after attending QuEST. As shown in Figure 2, the majority of teachers (70.8%) 
reported feeling somewhat confident in their ability to meet the needs of diverse learners prior to attending 
QuEST, although it is interesting to note that no teachers reported feeling not at all confident priot to QUEST. 
After attending QuEST,, all teachers reported feeling somewhat confident (29.2%) or very confident (70.8%) about 
their ability to meet the needs of diverse learners. Respondents were also asked to describe how UDL has 
influenced their science instruction for diverse learners. Many of the respondents indicated their exposure to 
the principles of UDL has made them mote cognizant of finding solutions that fit the needs of all students, 
not just struggling students. Teachers also reported they now have more resources available to them and have 
actively sought out others as well. Many teachers noted their assessment practices have changed based on the 
ptinciples of UDL, introducing a variety of ways to let students demonstrate their understanding of concepts. 
See Table 6 for representative comments. 


Educational Policy Improvement Center 29 


Evaluation of the QUEST DRK-12 Grant: Annual Report for Year 4 


How confident were 
you about your ability 
to meet the needs of 

diverse learners? 
n=32 


How confident are you 
about your ability to 
meet the needs of 
diverse learners since 
attending QUEST? 
n=32 


Teacher Confidence in Meeting the Needs of Diverse Learners 


Confidence levels for meeting the needs of diverse learners increased after the QUEST program. 


Very Confident 


Somewhat Not very 
Confident Confident 


Not at all 
Confident 


70.80% 12.50% 0.00% 


29.20% 0.00% 0.00% 


Figure 2. Teacher Perceptions of Confidence Related to Meeting the Needs of Diverse Learners. 


Table 6. Teacher Perceptions of How UDL Has Changed Their Instruction 


Representative Comments 


Providing 
solutions for all 
students 


Resources 


Alternative 
means of 
representation 
for assessment 


Lesson 
planning 


15 


| have been able to meet all students where they are in science. | have been able 
to group students based on reading level for science activities where reading is a 
prevalent piece of the experiment or activity. This has allowed students to all get 
the full experience of the content. 

| truly think | am proficient enough [to]Jallow my students to accomplish the same 
goals while completing different activities. The quest program has allowed me 
[to] help the diverse learners in a way that did not effect [sic] the learning of 
others and they all were able to achieve the same results. Due to this they were 
able to have group discussions and share their learning of science. 

| gained an abundance of tools and strategies. | also was introduced to more 
resources that | can apply almost immediately within my classroom. In my 
beginning years of teaching science it was very easy to give a struggling student 
something other than science to complete while we were doing an experiment. 
However, after going through UDL workshops two years in a row | have 
developed strategies to include these students. 

UDL has influenced my science instruction, as | am better able to assess my 
students and allow them to choose a way in which they can represent their 
thinking that best suites [sic] their needs. For example, | now provide several 
different means of representation when assessing students. Not all students are 
going to be able to demonstrate or explain their thinking using the same forum, 
so | use the UDL guidelines to ensure that | am accommodating to the needs of all 
students. This may mean that instead of demonstrating a concept on a piece of 
paper every time, students may have the option to make a video or PowerPoint 
or explain their thinking and understanding orally. 


| love having the resource of the UDL to use when planning lessons. It helps me to 
think about and plan for possible roadblocks that might interfere with lessons. 
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Representative Comments 


Although | was already using many of these strategies they were not formally 
identified as being part of the UDL. Many new ideas were also introduced to help 
me become more effective when addressing students with various challenges. 

| consider any accommodations or modifications that may be needed while in the 
planning stages of my science instruction and implement them during the actual 
lesson. 


Awareness of It makes me think of my particular group and make adjustments to my lesson 
and plan. It helps me stay accountable for making sure all needs are met. 
responsiveness | have been more open to what hurdles students may face. Then introduced to 


to barriers the various paths possible for these hurdles. 


Other lam constantly differentiating for and scaffolding students, but | am still not 
certain that | fully grasp UDL. 


Teachers were asked about their confidence in their ability to implement the 5E Learning Cycle 
before and after attending QuEST. As shown in Figure 3, many respondents reported being not at all confident 
(15.6%) or not very confident (34.4%) in their understanding of and ability to implement the 5E Learning Cycle 
ptior to attending QuEST. However, after attending QuEST, all respondents indicated that they were 
somewhat confident (34.4%) or very confident (65.6%) about their understanding of and ability to implement the 5E 
Learning Cycle. Respondents were also asked to provide an example of a change in their understanding or 
ability to implement the 5E Learning Cycle. Many of the respondents reported that the use of 5E helped to 
engage students in the content. Teachers also provided examples of how 5E helped to increase their 
effectiveness and confidence in science instruction, provided better organization and presentation of the 
science concepts, or changed their approach to pedagogy. Representative comments are presented in Table 7. 


Teacher Confidence in the 5E Learning Model 


Confidence levels for using the 5E Learning Model increased after the QuEST program. 


Very Confident Somewhat Not very Not at all 
Confident Confident Confident 
How confident were you about your 
understanding and ability to 
implement the 5E Learning Cycle 
prior to attending QuEST? fo 43.80% 34.40% 15.60% 
n=32 
How confident are you about your 
understanding and ability to 
implement the SE Learning Cycle 34.40% 0.00% 0.00% 


since attending QUEST 
n=32? 


Figure 3. Teacher Perceptions of Confidence Related to the 5E Learning Cycle. 
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Table 7. Teacher Perceptions of How the 5E Model Has Changed Their Instruction 


Representative Comments 


Increased | learned about the 5E learning cycle in Dr. Hanuscin’s class. | came to the summer 
effectiveness/ institute with prior knowledge, but | have solidified my thinking further through 
confidence the summer institute. For example, | now know ways | can implement assessment 
throughout the entire process of the design. We have talked about the ways we 
can help further student thinking in each stage. 
| had learned about the 5E model as a pre-service teacher. However, during that 
time, | found myself trying to get through all 5E’s in one class period, which is not 
too realistic. This Summer Institute taught me how to more effectively implement 
it into my classroom (over the course of a few days, etc) and how to make 
changes as | go, that will best meet the current needs of all of my students. 
| had no clue what the 5E Learning Cycle was prior to attending Quest. | feel | have 
a pretty good understanding of it now. 
| realized that | used the 5E instinctively without really knowing what the 5E was 
prior to the Summer Institute. Now | am more aware of the process and feel like 
lessons have improved because of my learned knowledge in this area. 
With this experience | have gained more and more confidence and feel like | know 
more of what my goals are in how | want to teach, but that will take time and 
experiences. Quest has done a fabulous job of letting me experience personally 
and then giving me the tools to teach using 5 E in lessons done extremely well. 
Now, | know much more what it should look like! 
Better | was able to understand how to allow the students to go through this process ina 
organization/ natural order. It was expected and understood by the students and it even 
structure of allowed them to ask questions and think about different processes of the cycle. 
instruction Some students would even know what the end result should be of what they were 
trying to accomplish and from there they knew what do achieve or stride for with 
the concepts. 
| have now modified the district provided curriculum to the 5E model so that my 
students are more engaged in creating their own learning experience versus being 
provided answers and then exploring. 
Change in | really like that it has changed who does all the problem solving basing it ona 
pedagogy more student-centered approach and getting the students more hands on 
concrete ideas. 
| learned that front loading vocabulary words are not necessary of this prior 
knowledge does not aid in understanding of materials presented. 
Use across | think using the 5E Learning Cycle is a way to incorporate the 5 cycles in all areas 
content areas of developing my lesson plans. | have found that | think about the 5E’s as |am 
developing plans in all areas. 


Other | learned about the 5E in college and participated in other professional 
development on it. 


Respondents were asked to describe something they have done within their classroom as a result of 
QuEST that they would not have done previously. The majority of teachers noted they increased their use of 
hands-on learning techniques. Teachers also indicated that they have increased their use of 5E and UDL as a 
result of QuEST, and they have introduced more small group instruction. 


Teachers were also asked to describe something they have done outside their classroom as a result 
of QuEST that they would not have done previously. Most teachers noted that QUEST allowed them to 
increase collaboration with other educators, within their school as well as across the state. Several teachers 
reported that QuEST facilitated changes in their personal knowledge and awareness of the subject matter or 
of their instructional practice. Three teachers indicated that they became involved in external science-related 


Educational Policy Improvement Center 32 


Evaluation of the QUEST DRK-12 Grant: Annual Report for Year 4 


activities because of their participation in QuEST. These activities occurring outside the classroom as a result 
of QuEST participation can be thought of as one type of project dissemination. Table 8 displays 
representative comments from the respondents. 


Table 8. Teacher Perceptions of Changes Within and Outside Their Classrooms as a Result of QUEST 


Participation 


Increased use 
of hands-on 
learning 


Use of 5E 


Increased use 
of small group 
instruction 


Use of a 
framework for 
instruction 


Increased 
emphasis on 
nature of 
science 


Increased use 
of UDL 


Increased 
collaboration 


Personal 
knowledge and 
awareness 


Representative Comments 


QuEST allowed me to teach physical science in a hands-on manner. | never 
imagined that | would use materials and experiments to instruct students how 
particles move and look. This is such an abstract idea and | always envisioned just 
telling my students the answer, however, QuEST introduced me to examples and 
experts that helped me in my planning process. 

QUEST provided me with materials to teach and time to think about science 
lessons. Both are such simple, but often overlooked, entities to teaching. 

Using the 5E method. | traditionally would do lessons straight from the textbook. 
While using the lessons from QuEST, my students were MUCH more engaged in 
the lessons than they would be with textbook and regular classroom activities. 
Letting the students work as a group to discover what | want them to learn. 
Instead of me feeding them the answers. Which is beneficial to the students in so 
many ways. 

Most of my past science teaching was done more in a whole group format. 
Through QuEST | saw the importance of students working in small groups and 
directing their own learning. 

It has helped me identify more of the backwards design & helping teacher 
implement it. 


Explain ideas of matter with greater clarity. | have also been more prepared to 
provide counter examples or additional experiences which would help students 
arrive at the objective without me "telling" them. 


| would not have given as much thought to UDL principles for all students. My 
experience with accommodations was aimed at students with identified needs 
only. |am more aware of doing this for all my students now. 


Engage and collaborate with teachers from other districts and learn from their 
experiences. 

Share the 5E model with other teachers and collaborate with them to help get 
out lessons into this format. 

The opportunity to interact and collaborate with my peers throughout the state 
has proved highly beneficial. | have been able to exchange ideas and learn about 
how other teachers are being successful in reaching their students. 

| have a new lens for thinking about different scientific concepts. | question way 
more than | used to. QUEST challenged me to think about science in new ways. 

| believe my questioning has increased. | have always been a curious person, but 
having questions asked of me that made me think at a deeper level, has been 
extended to so many areas that | have to stop myself sometimes from thinking 
and just doing. 

| think more scientific. 
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Representative Comments 


External | was able to participate in an ongoing science program that helped guide me to 
involvement in deliver instruction differently this year in my teaching. 
related Collaborate with my grade level on the same topic at the same time with matter. 
activities Last year, half of us taught it at the same time while others did not really teach it 
at all. This year, 4/5 of us taught it all throughout second trimester and with the 
same lessons, same materials, same progression, etc. ... So we could have 
shared conversations and refine our lessons and practices. 
e |’m not sure. 


Next, teachers were asked whether they had support from their school/district to implement what 
they had learned in QuEST. The majority of respondents indicated that they felt supported by their school 
and district. Additionally, respondents were asked whether they had support from their QuEST colleagues 
throughout the school year. Again, the majority of respondents indicated they also felt supported by their 
QuEST colleagues. Specific comments related to the ways in which teachers felt supported are included in 
Table 9. 


Table 9. Teacher Perceptions of School, District, and QUEST Colleague Support 


Representative Comments 


YES! My principal recommended it, as it was my first year of teaching. | was fully 
supported by him to implement what | learned at the institute in my classroom. 
My school was very supportive of the program. We were given the opportunity to 
share what we learned in one of our building meetings. The other teachers were 
receptive to the concepts because several of them have been through the process 
in other years with the QUEST program. They all wanted to know what we had 
learned so they could use some of the ideas in their classrooms. 

My principal and instructional specialist as well as my district science coordinator 
supported my participation in QUEST. They allowed me to use the 5-E learning 
cycle for plans as needed. 

Yes. Our science coordinator provided our grade level team with all of the 
additional supplies that we needed to teach the lessons the team wrote during 
QuEST. 

Not really. We are supposed to teach from the curriculum we are given and follow 
a specific pacing guide. My principal knew | was in the QUEST program so she 
allowed me to teach using the 5E method. | plan on using it in the future though 
because it is a great model. 

| wish that we had the support to actually teach the lesson, our principal has been 
supportive in implementing new ideas. We were not able to teach the lesson we 
needed because we due to inflexibility and resistance from that end. QUEST 
however has been amazing at answering questions and problem solving. 


Not Sure Not really. 


Yes 23 e Yes. Several of the other teachers in my building have been through the QUEST 
program in years past, so they were supportive of the program. Also, have two 
other team members in the building provided me the experiences | needed to 
grow asa teacher of science. 

e Yes, the other teachers from my school who also attended QuEST helped me with 
modifying our district curriculum to fit the 5E model. 
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Representative Comments 


Yes, | have been able to do a lot of collaboration with my grade-level team. | was 
also supported by the QUEST team. They were available to help when needed. 
YES! It was wonderful to be teaching this unit with my entire grade level. We were 
constantly bouncing ideas off one another and sharing the results on our students. 
Reflection and discussion was constant. 

Those from my district didn’t seem as motivated as | was to apply information in all 
areas throughout the year. 

To be honest, time to collaborate was not capitalized upon. Part of this was due to 
logistics. Every day, | was breastfeeding/pumping during my planning period and 
lunch. This limited my time with colleagues. However, | think conversations and 
support will continue and grow in the coming years. 


Preservice teachers were asked if they were able to apply what they learned through QuEST to their 
teaching of science. See Table 10 for the comments provided by three of the preservice teacher participants; 
two of the three responding preservice teachers reported being able to teach a unit on the properties of 
matter. 


Respondents were asked to describe the ways in which they were most affected by their participation 
in the QuEST program. A majority of the responses from respondents featured aspects of their pedagogical 
knowledge and/or understanding that had changed. Several teachers noted that they experienced increased 
confidence in their abilities as science teachers. Teachers also appreciated the opportunities for collaboration 
with other teachers, and several cited their increased content knowledge as a benefit of their participation in 
QuEST. Teachers were also asked what aspects of QUEST they would have liked to spend more ot less time 
on during the follow-up PD sessions. Teachers’ perceptions were varied, although many respondents noted 
that they would like to spend more time on lesson plans and less time on UDL. Table 11 includes teacher 
comments related to aspects of QuEST that they would have liked to spend more or less time on. Finally, 
teachers were asked to describe something they had learned in the follow-up PD sessions that contributed to 
their understanding or implementation of science instruction. Most teachers reported that the variety of 
instructional strategies gained from the follow-up sessions were the most useful in their classrooms. Many 
also noted that they developed a better understanding of the use of assessment practice. Other responses 
were equally divided among the 5E model, the increased content area understanding, and the principles of 
UDE. 


Table 10. Preservice Teachers’ Use of Knowledge and Skills Learned Through QuEST in Teaching Science 
Comments 


e Yes, | am student teaching in the fifth grade, so | had the chance to implement 
what we learned during the summer institute. The prior knowledge learned from 
the summer institute helped me question my students’ thinking. | was able to take 
what was learned in the summer and apply it to the classroom experience. | had a 
better understanding to answer student questions. 

Yes! In my actually teaching of science, | was in second grade instead of fifth grade, 


but | found myself still relating back to my QuEST experience. | found myself 
thinking of myself as a learner first when planning instruction. | worked on not just 
giving my students the “correct” answer and allowed them time to explore it on 
their own first. 

Since |am a preservice teacher, | did not get to apply what | learned directly to a 
classroom, but | was able to adapt what | learned and apply it to my field and my 
job at an after school program. 
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Table 11. Teacher Perceptions of Various Aspects of QuEST Participation 


Changes in 
pedagogy 


Increased 
confidence 


Collaboration 


Increased 
content 
knowledge 


Lesson 
development/ 
planning 


More 
collaboration 


More 
demonstrations/ 
hands-on 
activities 


None 


Teach lessons 


13 


Representative Comments 


It helped me better understand the conceptual storyline and make sure my 
lessons within a unit flow together. 

Gaining knowledge and confidence for what it looks and feels like to teach 
science in a more effective manner, student centered and question based. As 
well as more practice with the 5 E's that can be used cross-curricular. 
Differentiation for students—I was able to better understand ways to change the 
instruction for various types of learners. 

My confidence to know as a teacher | was feeding my students the science they 
needed to prepare them for additional courses. 

| feel stronger in teaching science than | ever have been. | enjoyed the materials 
we were given and the way the lessons were set up. 


One of the ways | was most affected by participating in this program was just the 
interaction with teachers and professionals from various backgrounds, 
collaborating, working together to discover better and more effective ways to 
teach a subject that is often put on the back burning [sic] but so necessary to 
open up the understanding of students. 

| felt as though | had a support team. It truly made a difference in my instruction. 
My own conceptual knowledge increased as a result of the program. After the 
program, it challenged me to rethink how | planned lessons in other units of 
science. 

| have gained a deeper understanding of matter and am much more confident 
and excited to teach the subject. Also, | understand that students will not come 
to the answer of a big question right away, instead, build up to a logical 
conclusion through discovery. 

My love for science has grown tremendously. | have been doing research outside 
of work to challenge myself to learn new things. 


Creating more science lessons that fit with the standards and followed 5E. We 
did this more as the sessions progressed due to the QuEST leaders listening to 
our feedback after sessions! ! 

Planning and implementing the strategies into other pieces of our science 
curriculum. 

Planning and more activities that would help us delve even deeper in the 
content. 

Seeing other 5E cycles. 

Doing more collaboration. 

| would like time to think through more planning with my entire team. 


Science experiments. | learned so much from the experiments and the hands- 
on learning is what | will be able to take back to my class for years to come. 
More hands on during the extended yearlong PD sessions. 


Not sure but the time that was spent was well spent! 


| would have liked to teach the lessons. | was not a part of the group that got 
to. 
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Representative Comments 


e ~=Anything that we hadn’t already done many times in the summer program. The 
PD sessions were a waste of our valuable time. 


Talking about the UDL. | feel it was a bit overstated. | know it’s important but 
would rather have designed lessons with materials ready to go in the weeks 
ahead. 
Spending so much time with UDL. | felt like we went through problem solutions 
so much it became over generalized and wasn’t as helpful. 

Repetition/ Reviewing what we did over the summer. 


review 


Other Talking about how to run our classroom. 


Nothing/ | enjoyed and benefited from everything we did in the follow up sessions. 
not sure | cannot think of anything. The learning [was] interesting and fun. 


General For me, it was the structures. So the eBook, or hyperdoc, that Dr. H made with 
instructional structures and new ideas to think through things was helpful. For example, 
strategies right after the session | attended, | used the Frayer model in class AND it 
informed my instruction! 
| learned a lot, but most of all, receiving resources, links, and listening to others 
has helped me research and discover even better ways to help my lessons 
become more meaningful to my students. 
More in-depth about previous conversations we have had (ex.: following up on 
using models as a way to explain thinking) and brainstorming several different 
scenarios in applying these analyses. 

Assessment | learned how there isn’t always one correct answer and if the student has an 
understanding and can provide how they found their idea then that 
demonstrates a level of understanding. 

How to question students. 

| learned some new strategies to implement in the elaborate and evaluate 
phases that | would not have thought of if left to my own devices. 

That students need to explore. The exploration phase is so important and my 
students were missing out on that until | went to QUEST. 

More ways to help struggling students. 

Looking up the UDL online and it’s [sic] suggestions was nice. A couple of the 
follow up had some hands on materials and lessons that | used. 

Content | have learning a deeper knowledge of matter, and different ways | can 

knowledge implement it in my classroom. Each follow up sessions has allowed me to go 
back and teach something differently every time. 
| learned how to build on the matter unit by incorporating the magnets into the 
lesson plan. How to use different models into the lesson plans. 

Collaboration That more heads are better than one. Loved the collaboration. 


Other It has been a really long time ago and | cannot remember just one specific thing 
from that follow up session. More of the same information from the previous 
answers. 
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Research Plan Implementation Evaluation 


This section provides a summary of the findings from the evaluation activities related to the 
implementation of the QuEST research plan. 


Research Plan 


The QuEST project research plan includes the annual recruitment of schools within the partner 
districts for participation in the project intervention and four phases of data collection and analysis. Research 
plan evaluation consisted of monthly meetings with principal investigators, regular review of progress on the 
project management tool Asana, and review of relevant project materials, including curricular materials and 
summaries of teacher- and student-level data. An end-of-summer evaluation report was provided to the 
QuEST project staff in the fall of 2016 to summarize data collection activities that occurred during the 
summer institute and interviews with key staff members after the summer institute. Lessons learned were 
shared with the QuEST staff through both formal and informal means, allowing a continuous process of 
feedback and refinement of research processes. As in the previous project year, the main challenge faced by 
the QuEST project around the research plan was in the recruitment of teachers/schools. QUEST project staff 
members were able to make adjustments to the research plan by recruiting for project participation at other 
schools and districts to replace those who dropped out or experienced teacher reassignment that affected the 
availability of teachers to participate. The QuEST staff continued to refine their recruitment procedure to 
best facilitate the process. 


Baseline School and Student Achievement Data 


This section contains a summary of demographic data (see Table 12) and student achievement data 
(see Table 13) for participating treatment schools. As this is a multiyear and longitudinal study, the final 
report will include summative analyses of achievement data collected and examined by QuEST and EPIC 


project staffs. 


Table 12. Total Enrollment and Student Demographic Information by School, 2016 


District 


Columbia 93 


Columbia 93 
Columbia 93 
Columbia 93 
Fulton 58 

Fulton 58 
Hazelwood 
Hazelwood 
Hazelwood 
Independence 30 
Independence 30 


Independence 30 


Normandy 


Normandy 


Educational Policy Improvement Center 


School 


Mary Paxton Keeley 
Elem. 


Rock Bridge Elem. 
Shepard Blvd. Elem. 
Thomas Benton Elem. 
Bartley Elem. 
McIntire Elem. 

Cold Water Elem. 
Garrett Elem. 

Lawson Elem. 
Fairmount Elem. 
Procter Elem. 


Thomas Hart Benton 
Elem. 


Barack Obama Elem. 


Lucas Crossing Elem. 
Complex 


Am. Multi 
Hispanic Indian -race 


Pacific Free/Reduced 


Enrolled Asian Black Islander White Lunch 
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Am. Multi Pacific Free/Reduced 
District Enrolled Asian Black Hispanic Indian -race Islander Lunch 


North Callaway Hatton-McCredie Elem. 
R-1 


North Callaway Williamsburg Elem. 
R-1 


St. Louis City Buder Elem. 
St. Louis City Hodgen Elem. 


St. Louis City Shaw Visual/Performing 
Arts Center 


St. Louis City Woodward Elem. 289 73% 


*Indicates percent has been suppressed due to a potential small sample size. 


Table 13. Percentage of Students Performing Below Basic in Key Subject Areas on the MAP by School in ELA, 
Math, and Science (Grade 5), 2016 


District 


Columbia 93 
Columbia 93 
Columbia 93 
Columbia 93 
Fulton 58 

Fulton 58 
Hazelwood 
Hazelwood 
Hazelwood 
Independence 30 
Independence 30 
Independence 30 
Normandy 
Normandy 

North Callaway R-1 
North Callaway R-1 
St. Louis City 

St. Louis City 

St. Louis City 

St. Louis City 


Below Basic 
School Math Science 
Mary Paxton Keeley Elementary 
Rock Bridge Elementary 
Shepard Blvd. Elementary 
Thomas Benton Elementary 
Bartley Elementary 
Mcintire Elementary 
Cold Water Elementary 
Garrett Elementary 
Lawson Elementary 
Fairmount Elementary 
Procter Elementary 
Thomas Hart Benton Elementary 
Barack Obama Elementary 
Lucas Crossing Elementary Complex 
Hatton-McCredie Elementary 
Williamsburg Elementary 
Buder Elementary 


Hodgen Elementary 


Shaw Visual/Performing Arts Center 


Woodward Elementary 


*Indicates no data available or cell value is too small. 
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Conclusions and Recommendations 


In keeping with the responsive, utilization-focused evaluation practice, conclusions and 
recommendations around the implementation of the three focal areas of the QUEST project have been 
shared with the QuEST team in an ongoing process during the project year. 


The following conclusions and recommendations about the data collected during the Year 4 
evaluation of the QuEST project are categorized by the three foci of the overall evaluation plan. 


Professional Development Implementation Evaluation 


Professional Development Observations 


Summer institute. Based on the observation protocol of professional development standards, the 
QuEST professional development program seemed to adequately address all nine standards for professional 
learning. Hands-on activities and the opportunities for participants to share experience and knowledge were 
evident in all observed sessions. 


Standards where support evidence was missing were in Practice during Week 1 of the institute and 
Selection and Organization of Content, Research, Community of Learners, Diversity, and Practice in Week 2. This may 
have to do with the fact that the evaluators were not able to observe every PD session or that the design of a 
series of sessions is structured in such a way that evidence is not readily available. However, the project 
leadership may want to reflect on whether any of the indicators that were not addressed could be important 
additions to the quality of the PD, or whether they are simply outside the PD model used for QuEST. 


Saturday Follow-up Sessions. Based on the observation protocol of professional development 
standards, the QuEST professional development program seemed to adequately address all nine standards for 
professional learning during the observed Saturday session. As was observed during the summer institute, 
hands-on activities and the opportunities for participants to share experience and knowledge were evident in 
all observed sessions, and well received by the teacher participants. Activities included whole group, small 
group, and individual work arrangements. 


Standards where support evidence was missing were in Se/ection and Organization of Content and Practice. 
The project leadership again may want to reflect on whether this is a gap to be addressed, a reflection of only 
observing one out of a series of four session, or a philosophical difference in approach. 


Teacher Focus Groups 


Teacher participants were generally very satisfied with the program’s attention to the following 
aspects of professional development: materials, delivery and organization, facilitator quality, applicability, and 
overall experience. Participants reported that the professional development experience was more useful than 
other types of professional development they had attended. They also expressed high levels of satisfaction 
with the science content gained from the instructional modules and the ability of the facilitators to model 
deeper questioning skills. 


A few participants noted issues of dissatisfaction with the program. When asked to provide 
suggestions for improvement, comments focused on mental exhaustion during the institute, concern about 
implementing newly learned strategies and lessons in light of district-specific limitations, and requests for 
increased planning time with individual materials. 


Project Staff Interviews 


QuEST project staff members reported that the team continued to function efficiently and 
effectively in meeting the project goals. Success was attributed to the continued high levels of communication 
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and organization, and to staff members’ increased confidence with the materials both within and outside their 
areas of expertise. 


The biggest challenges faced by the project team members were issues with attrition during the 
recruitment phase, space issues during Week 1 that limited the involvement of the QuEST staff in the 
content lesson time, and the loss of the project manager in the spring. 


Teacher Survey 


The results of the 2016 spring survey indicated teachers’ perceptions of their level of qualification to 
teach science, physical science, and properties of matter had increased from the time when they completed 
the project enrollment survey. Most notable was the percentage of teachers reporting feeling wel/ qualified to 
teach properties of matter (62.5%) after participating in the QuEST project, up from 10.5% reporting feeling 
well qualified to teach this subject matter at the time of enrollment. Teachers reported an increase in their 
perception of their ability to meet the needs of diverse learners due to their participation in QuEST, with 
87.5% of teachers reporting they were somewhat confident or very confident prior to their participation, compared 
to 100% of teachers reporting that they were somewhat confident or very confident after their participation. All 
teachers (100%) reported feeling somewhat confident or very confident about their ability to implement the 5E 
learning cycle after participating in QUEST, compared to only 50.1% of respondents reporting feeling 
somewhat confident or very confident about their ability to implement the 5E learning cycle before participation. 


Many teachers reported an increase in confidence in teaching science and in teaching and 
implementing new strategies in their classroom in general after their participation in the project. Additionally, 
respondents noted that their way of thinking about science had been affected as well. Other aspects of 
QuEST participation that most affected teachers included increased pedagogical knowledge and/or skills, the 
opportunity to regularly collaborate with other teachers, and increased science content knowledge. Teachers 
reported that the follow-up PD sessions contributed to their understanding or implementation of science 
instruction through increased exposure to and practice with instructional strategies, including the 5E model 
and UDL, and an increased understanding of the use of a variety of assessment techniques. 


Recommendations 


Although it is unknown whether there will be future QUEST (or QuEST-like) professional 
development opportunities, the following recommendations are provided: 


e Build on the use of themes that provide through lines in the curriculum to which participants can 
refer as they progress through the materials. 


e Consider starting with UDL instruction in afternoon sessions during Week 1, setting a foundation for 
the idea that UDL strategies connect to all the other strands. 


e Consider whether to test an alternative delivery of Week 1 content. For example, try presenting 
science content in the mornings of Days 1, 3, and 5, but in the afternoons of Days 2 and 4. 


e Provide more opportunities for preservice teachers to demonstrate their knowledge and to connect 
to current teacher participants. 


e Emphasize objectives of each session, when appropriate, and provide recaps of the previous day 
when possible, including time to answer questions brought up in the previous day’s checkouts. 


e Continue to hold pre-institute planning sessions/research retreats and use them as a space to pilot 
activities and involve non—content area staff members. 


e Continue to encourage teachers to take risks and learn to be comfortable with not knowing the 
“correct” answer. 
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e Provide more opportunities for teachers to explore application of the institute content and skills in 
other disciplines in which they may feel more comfortable than science. 


e Continue and increase the amount of time that teachers have for collaboration and planning during 
the Saturday follow-up sessions. 


e Continue the exploration of best practice in science instruction by providing more resources around 
topics of teacher interest, such as assessment, technology, and hands-on experiment ideas. 


e Decrease the amount of time spent on topics that have already been thoroughly covered in previous 
sessions. 


Research Plan Evaluation 


Based on the ongoing communication with the QuEST project staff and use of the project 
management tool Asana, the QuEST research plan implementation seemed to proceed according to plan. 
Data collection and analysis activities were carried out in a timely and organized fashion. Smoother operation 
of the research components during the institute occurred due to increased communication and planning. 
Changes in/additions to the project staff were accomplished without major disruption to the working of the 
team as a whole. Recruitment and attrition were again the main challenges encountered by the QuEST team, 
requiring slight modifications to the research plan. 


Recommendations include continuing the process put into place to ensure timely and thorough data 
collection and analysis activities; continuing to hold ongoing research team meetings; continuing the use of 
the project management tool; and maintaining a high degree of communication within and between project 
teams to ensure coordination of research activities. As the research team synthesizes several years’ worth of 
data, the evaluation team will conduct reviews of the analysis plans and outcomes for the summative report. 


Dissemination Plan Implementation Evaluation 


QuEST researchers accomplished the dissemination of project information and results to the public, 
researchers, and teacher educators in a variety of ways. Project evaluators continued to collect information 
and artifacts relating to dissemination during Year 4 of the QuEST project. A discussion of the avenues of 
dissemination across all project years will be summarized and included in the summative evaluation report. 
Recommendations at this stage in the project include continuing to reach out to QuEST alumni and other 
local networking activities to help in the dissemination process. 


Year-to-Year Comparison 


From their own observations and conversations with participants, evalautors noted the following 
changes between aspects of the 2015 and 2016 summer institutes: 


e Use of new reflection forms designed to elicit more specific thoughts and feedback, resulting in an 
increased quality and depth of reflections across the summer institute. 


e Increased participant interest in developing and integrating formative assessment. 
e Increased acceptance of the Treatment 1/Treatment 2 division during Week 2. 


e Increased organization and objectives for the Week 2 Treatment 2 activities, and flexibility in 
responding to individual/group needs. 


e Increased participant understanding of the concept and use of the conceptual storyline and UDL. 
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Appendix 


QUEST Professional Develooment Observation Tool 


Observation Data 


PD Implementation (Goal 1): The evaluation question for this component asks to what extent has the 
QuEST project achieved its goal within the expected timetables using the stated principles and processes 
(summer PD, situated PD experience, quarterly follow-up sessions). The evaluator will examine the factors 
that facilitate or hinder implementation plans as well as the way in which the project staff manages and 
overcomes battiers. 


General Details 


Date: Observer: 
Group: Lesson Topic: 
Start Time: End Time: 


Length of Observation (in minutes): 


Participant Information 


Number of Participants: Number of Facilitators /Trainers: 


Classroom Organization 


Which types of participant organization were observed during the observation? (check all that apply) 


Whole class Small groups Individual work 


What was the overall level of participant involvement? (check only one) 
Low attention (less than 25% on task throughout observation) 
Moderate attention (25% to 75% on task throughout observation) 


High attention (more than 75% on task throughout observation) 
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Standard 


Selection and 
Organization of 
Content 


Clearly defined 
learning objectives 
provide a focus for 
the content, which 
is organized 
around a 
framework or 
structure 


Standard 


Reseatch 


Content reflects 
current research, 
sound theory, and 
craft knowledge as 
appropriate 


Support Evidence for Addressing the Standard 


Clearly stated the purpose of the week’s activities 
Observed. Not Observed. 


Defined the target audience and necessary prerequisite skills 


Observed. Not Observed 


Solicited and answered questions about their instructional 
goals/activities from teacher participants where necessary 


Observed. Not Observed 


Clearly defined expected outcomes 


Observed. Not Observed 


Formulated a limited number of goals and/or objectives 


Observed. Not Observed 


Developed a conceptual framework to highlight major ideas 
to be presented and to organize the content 


Observed. Not Observed 


Support Evidence for Addressing the Standard 


Described the research, evidence, theory, or reports from 
practitioners that supported the content 


Observed. Not Observed 


Avoided conjecture and lots of personal opinion, but 
included valid alternate opinions to help the participants 
make their own judgments 


Observed. Not Observed 


Fully cited all references and research 


Observed. Not Observed 


Identified professional standards or content standards where 
applicable (e.g., CCSS, NGSS, GLES) 


Observed. Not Observed 


When possible, cited evidence of effectiveness of practices 
presented in the training or presentation 


Observed. Not Observed. 


Suggested Practices 


Written learner 
objectives 
communicated at the 
beginning of 
instruction 


Graphic organizer(s) 


Outline 


Assess learners’ prior 
knowledge and 
experience (formally 
or informally) 


Suggested Practices 


Report results of 
research studies 


Describe previous 
work with QUEST 


Consult content 
standards 
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Standard 


Principles of 
Adult Learning 


The learning 
activities ate 
consistent with 
principles of 
adult learning 


Standard 


Community of 
Learners 


The design for 
the professional 
learning 
experience seeks 
to create a 
community of 
learners 


Support Evidence for Addressing the Standard 


Drew upon and honored learners’ prior knowledge and 
experience 


Observed. Not Observed. 


Provided opportunities for learners to connect new learning 
to their own work 


Observed. Not Observed. 


Designed and used problem-centered activities 


Observed. Not Observed. 


Provided strategies, tools, and techniques that participants 
could begin to use immediately 


Observed. Not Observed. 


Offered opportunities for participants to assume 
responsibility for their own learning (i.e., to be as self-directed 
as appropriate given content) 


Observed. Not Observed 


Support Evidence for Addressing the Standard 


Encouraged dialogue and sharing among participants 


Observed. Not Observed. 


Encouraged dialogue and sharing between participants and 
PD staff/content experts 


Observed. Not Observed 


Incorporated opportunities for team building and 
collaboration during sessions and in designed follow-up 


Observed. Not Observed 


Considered structures and practices that connect participants 
across time 


Observed. Not Observed 


Encouraged risk taking 
Observed. Not Observed. 


Suggested Practices 
Assess learners’ prior 
knowledge and 
experience (formally 
ot informally) 


Activate prior 
knowledge 


Build in time for 
individual reflection 


Develop case studies 
and/or scenarios for 
problem solving 


Offer hands-on 
practice in using 
related tools, 
techniques, and 
strategies 


Connect new 
strategies directly to 
teacher experience 


Encourage 
participants to set 
their own 
improvement goals 


Suggested Practices 
Plan carefully for 
grouping and re- 
grouping of 
participants 
Collaboration in 
small groups 


Introduce norms for 
participation to the 


group 


Use cooperative 
learning strategies 


Investigate use of 
online community 
for past and present 
participants 
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Standard 
Practice 


The design 
affords 
participants 
opportunities to 
practice new 
skills and 
includes 
appropriate 
follow-up 
activities 


Provided for celebration of small successes and learning from 
failures 


Observed. Not Observed 


Support Evidence for Addressing the Standard 


Incorporated opportunities for participants to practice new 
skills in a safe workshop environment and to receive feedback 
from facilitator and colleagues 


Observed. Not Observed 


Facilitated student teaching component in a clear and 
organized manner 


Observed. Not Observed. 


Encouraged the use of innovative and/or novel approaches 
where appropriate 


Observed. Not Observed 


Ensured adequate time for lesson planning and team building 
before each teaching session 


Observed. Not Observed. 


Provided guidelines for analyzing student work and using data 
to guide future instruction in line with PD goals and activities 


Observed. Not Observed. 


Included debrief time with content experts and PD facilitators 
to provide feedback around daily instruction, including 
content, process, and addressing student misconceptions 


Observed. Not Observed 


Designed follow-up activities to support teachers in new 
content and process 


Observed. Not Observed 


Included structures to track participants following the 
professional development 


Observed. Not Observed 


Encouraged peer classroom observations when appropriate 


Observed. Not Observed 


Developed capacity of individuals at site to provide leadership 
for follow-through 


Observed. Not Observed. 


Provide participant 
lists 

Build in 
opportunities for 
individual and group 
reflection on group 
processes 


Suggested Practices 


Use role plays, 
rehearsals, etc. 


Adapt critical friend 
approach to 
workshop setting 


Encourage support 


groups 


Online follow-up 
through discussion 
areas, listservs, of 
other social media 


Self-paced follow-up 
activities that are not 
monitored by a 
facilitator 


Encourage 
participants to set 
their own 
improvement goals 
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Standard 
Diversity 


The content and 
activities ate 
sensitive to and 
respectful of 
cultural diversity 


Standard 
Process Design 


The design for 
the learning 
experience 
includes an 
appropriate 
balance between 
the presentation 
of content and 
active participant 
engagement 


Support Evidence for Addressing the Standard 
Established norms of respect, openness, and listening through 
both the content and process 


Observed. Not Observed 


Used language and media elements that were respectful of and 
appropriate to all 


Observed. Not Observed 


Planned for an inclusive approach (i.e., used strategies that 
engaged all individuals and subgroups) 


Observed. Not Observed 


Selected resources that brought diverse points of view to the 
table 


Observed. Not Observed 


Provided opportunities for educators to share knowledge, 
skills, and strategies for involving families or other 
stakeholders appropriately 


Observed. Not Observed 


Support Evidence for Addressing the Standard 
Incorporated a variety of presentation strategies and activity 
formats 


Observed. Not Observed 


Included media elements that appropriately supported the 
content 


Observed. Not Observed 


Ensured that participants have opportunities to develop or 
review related knowledge base 


Observed. Not Observed 


Varied learning activities, interspersing didactic with active, 
hands-on learning 


Observed. Not Observed 


Designed for all learning styles: the visual, auditory, kinesthetic 
Observed. Not Observed 


Structured daily activities and kept on task so that adequate 
time was available for instruction, activities, and reflection 


Observed. Not Observed. 


Suggested Practices 
Use language and 
media that are free 
from bias 


Include media 
elements of various 
genders, races, ages, 
and abilities 


Suggested Practices 
Balance lecture of 
text-based 
presentation with 
hands-on ot 
reflective activities 


Provide 
opportunities for 
various intensity and 
mastery levels of 
activities 
Continuously scan 
and monitor to assess 
extent of participant 
engagement 


Use warm-ups and 
energizets to energize 
participants at critical 
points 


Provide transitions to 
maintain the flow of 
activities 

Use schedule/timing 
so that event and 
participants stay 
focused 
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Standard 


Self- 
Assessment 


The learning 
experience 
includes formal 
and informal 
opportunities for 
self-assessment 
that facilitate 
participant 
evaluation of 
their current 
practice, 
willingness to 
change, and 
monitoring of 
their progress 


Evaluation 


Participants will 
have 
opportunities to 
evaluate the 
quality of the 
professional 
learning 
experience 


Support Evidence for Addressing the Standard 


Assured that participants were clear about expectations for 
their own learning and change 


Observed. Not Observed 


Allowed for practice and assessment of all stated outcomes 


Observed. Not Observed 


Encouraged participants to identify and build on their 
individual strengths 


Observed. Not Observed. 


Provided opportunities for participants to identify barriers to 
change 


Observed. Not Observed 


Built in debriefs at critical points in the training (e.g., at the 
end of a day) 


Observed. Not Observed 


Sought participant feedback throughout the formal learning 
experience — both formally (e.g., quick feedbacks at day’s end) 
and informally (e.g., by asking participants to signal how it’s 
going for them at intervals throughout the learning 
experience) 


Observed. Not Observed. 


Ample time scheduled for end-of-session evaluation 


Observed. Not Observed 


Suggested Practices 


Develop and 
facilitate participant 
use of rubrics, 
checklists, etc. 


Introduce 
participants to 
concepts of feedback 
using some language 
that focuses on 
positive vs. areas for 
improvement 


Model reflective 
practice to 
participants 


Provides 
opportunities for 
participants to 
surface and examine 
attitudes and beliefs 
that relate to 
proposed behavioral 


changes 
Standard Support Evidence for Addressing the Standard Suggested Practices 


Be open to 
participant questions 
and concerns 


Plan for brief written 
feedback at the end 
of each day of a 
multiday event 


Use feedback to 
identify and meet the 
needs of participants 


Consult with staff in 
design of follow-up 
evaluations 
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QUEST Teacher Focus Group Protocol —- Summer 2016 


The evaluation question for this component of the research project asks to what extent has the QUEST 
project achieved its goal within the expected timetables using the stated principles and processes (summer 
PD, situated PD experience, quarterly follow-up sessions). The evaluator will examine the factors that 
facilitate or hinder implementation plans as well as the way in which the project staff manages and overcomes 
barriers. 


The primary objectives of the QUEST PD program are to: 
e Strengthen teachers’ knowledge of physical science content appropriate to the K—6 curriculum; 
e Enhance teachers’ pedagogical knowledge for teaching elementary science; 
e Support the development and enactment of teachers’ pedagogical content knowledge; and 


e Improve the learning of all students in K—6 science. 


The goals of the teacher focus groups are to: 
e Gauge teacher satisfaction with professional learning; 
e Determine whether teachers mastered new knowledge and skills; and 


e Gather teacher perceptions around how applicable the new knowledge and skills will be to their 
classroom. 


Welcome 


Thank you for your willingness to participate in this focus group and to share your expert insights into the 
professional development you are receiving in this summer institute. (Introduce Kristine and self.) We are 
here from the Educational Policy Improvement Center in Eugene, Oregon, as observers of this workshop. 
We serve as the evaluators of the QuEST project, which means that we are supporting the project leaders in 
determining improvements to the project and, later on, we will look at the project’s outcomes. 


Our discussion today will focus specifically on your perceptions of the usability of the materials presented to 
you in this PD experience and on your level of overall satisfaction with the PD. 


The results will be used to improve next year’s summer institute. 


You were selected because you are participating in QuEST as (change this depending on whether talking to 
Treatment/Comparison/Preservice teachers). 


Introductions 


Please tell us your name, where you teach and at what grade level (change this when speaking to preservice 
teachers — where you are in your preservice program and the certification level you are working toward), and 
why you signed up for this PD experience. 


Guidelines 


My role as facilitator will be to guide the discussion. There are no right or wrong answers, only differing 
points of view. Please feel free to share your point of view even if it differs from what others have said. Keep 
in mind that we’re just as interested in critically constructive comments as positive comments, and at times 
the constructive comments are the most helpful. 
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We will be audio recording this discussion today. I will be asking each one of you for your consent to do so. 
Say to the participant: This is (evaluator) interviewing (participant) on (date). We are 
audio recording this interview. Is that ok with your (Wait for positive response.) Thank you. 


We ask that you silence your phones. If you cannot and if you must respond to a call, please do so as quietly 
as possible and rejoin us as quickly as you can. 


Discussion Topics 


Week 1 


We’re going to start by asking you some questions about last week, which was Week 1 of the summer 
institute. 


Communication of Objectives 


Did you feel like you understood the overall objectives of the QuEST project? 


Were the objectives for the first week of the PD institute clearly outlined? 


Organization and Delivery of Material 
What would you say about the quality of the materials used in the PD? 


How well organized was the delivery of the PD material in the first week? 


Facilitator Quality 
How well prepared were the instructors to facilitate the workshop? 
What would you say about the ability of the facilitators to promote understanding of the PD 


materials? 


Applicability to Your Classroom Instruction (PD, curriculum, 5E model, UDL) 


How easily do you think the physics curriculum will integrate into your classroom? 
How easily do you think the 5E model of science instruction will integrate into your classroom? 


Do you have any concerns about integrating the 5E model and/or the science content into your 
classroom? 


How easily do you think the principles of the Universal Design of Learning will integrate into your 
classroom? 


Do you have any concerns about integrating the principles of the Universal Desien of Learning into 
y' y g g Pp Pp s & 
your classroom? 


How well prepared do you feel to use what you have learned in your classroom (science content, 
ConcepTest)? 


How effective do you think the 5E model will be in promoting students’ understanding of science? 


What would you say about the overall delivery of the PD in the first week? (timing, transitions, 
asking /answering questions, etc.) 
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Week 2 
Now, we’d like to talk to you about the activities that have occurred this week, which is Week 2 of the 


summet institute. 


How Well Were Objectives Communicated 


Were the objectives for the second week of the PD institute clearly outlined? 
Organization and Delivery of Material 

How well organized was the delivery of the PD material in the second week? 
Facilitator Quality 

How well prepared were the instructors to facilitate the workshop? 


What would you say about the ability of the facilitators to promote understanding of the PD 
materials? 


Situated PD Experience (Tx/Preservice Teachers Only) 


How would you describe your experience in the embedded teaching experience in Week 2? How has 
this experience differed from other types of professional development you have attended? 


Do you feel like the embedded teaching experience will allow you to better implement what you 
learned in your classroom this year/when you have your own classroom? 


PD Experience (Comparison Only) 
How would you describe your experience in the second week of professional development? 


Do you feel like the activities in the second week will allow you to better implement what you learned 
in your classroom this year? 


What would you say about the overall delivery of the PD in the second week? (timing, transitions, 
asking /answering questions, etc.) 


Wrap-Up 
We have two final “finish the sentence” questions. Let’s go around the room and finish each. 
Elements of this PD experience from which I learned most were... 


Elements of this PD experience that could be improved wete... 


Again, thank you for your contribution to this focus group. We have enjoyed spending time with you today 
and hearing about your participation in this professional development experience. 
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QUEST Spring Teacher Survey 2016 


Consent 


The Educational Policy Improvement Center (EPIC) is evaluating the QuEST grant to investigate the 
effects of the professional development model used to provide teachers with support around 
elementary science instruction. EPIC staff, along with staff from QuEST who lead the grant, have put 
together a survey to ensure we have teacher input on the impact of the professional development 
received as part of the QuEST project during the summer institute of 2015 and the following 
academic school year. This survey should take about 20 minutes to complete. 


Use of Teacher Survey Data 


The use of teacher survey data by QUEST may include, but is not limited to, overall reporting of 
results at the school level to the grant funding agency; use of school-level and grant-level results to 
improve professional development for teachers around elementary science instruction; and sharing 
overall results with others in education who want to learn more about the QUEST model of 
professional development delivery. 


Confidentiality 


EPIC and QuEST will take all steps necessary to make sure that the results of this survey are kept 
confidential. Surveys are collected and delivered to EPIC for statistical analysis. There is always the 
possibility of tampering from an outside source when using the Internet for collecting information. 
While the confidentiality of your responses will be protected once the data are downloaded from 
the Internet, there is always the possibility of hacking or other security breaches that could 
threaten the confidentiality of your responses. Please know that you are free to decide not to 
answer any question. 


Terms of Assent 


Completing this survey is voluntary. If you decide to complete the survey, you are free to stop at any 
time. EPIC and QuEST do not believe there are any risks related to your participation and hope that 
your survey responses will help the grant team to improve teacher professional development 
around science instruction. If you have any questions regarding the survey or the evaluation of this 
grant, you may contact the evaluator, Tracy Bousselot, at 541.246.2600 or 877.766.7729 or 
tracy_bousselot@epiconline.org. 
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Please describe how science instruction was accomplished in your school in the 2015-16 
school year. 


Do you use a published curriculum (e.g., kit or textbook) or is the science curriculum teacher- 


developed? 
© Commercially offered curriculum from a publisher 


© District-created curriculum 
©. Teacher-created materials 
O 


Other, please explain: 


How often does science instruction happen in your classroom? 
Oo. Daily 
Several times per week 


@) 
©. Once weekly 
O 


Less than once weekly 


What is the typical lesson duration of the science instruction in your classroom? 


© 30 minutes or less 
© 31-45 minutes 


© More than 45 minutes 


In the 2015-16 school year, did you teach your students about electrical circuits and energy? 
©. Yes; a full unit 


©. Yes; several learning cycles 
©. Yes; only one learning cycle 
Oo No 


Kk 
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How much total instructional time (in minutes) was spent teaching students about electrical 
circuits and energy? 


How close were your lessons to the electrical circuits and energy unit that you experienced in 
the Summer Institute? 
© Very close 


© Close 


©. Not very close 


Since your participation in the 2015 QUEST Summer Institute: 


How qualified do you feel to teach science, in general? 
© Well-qualified 


©. Adequately qualified 
O. Not well-qualified 


How qualified do you feel to teach physical science topics, in particular? 
© Well-qualified 


© Adequately qualified 
©. Not well-qualified 


How qualified do you feel to teach electrical circuits and energy? 
© Well-qualified 


©. Adequately qualified 
©. Not well-qualified 


How has your instruction been influenced by your experience as a learner of science in the 
summer institute? 


How confident were you about your ability to meet the needs of diverse learners prior to 


attending QuEST? 
© Very confident 
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© Somewhat confident 
© Not very confident 


© Notatall confident 


How confident are you about your ability to meet the needs of diverse learners since attending 
QuEST? 
©. Very confident 


© Somewhat confident 
© Not very confident 
O 


Not at all confident 


How has UDL influenced your science instruction for diverse learners? Please provide a 
concrete example. 


How confident were you about your understanding of and ability to implement the 5E Learning 


Cycle prior to attending QUEST? 
©. Very confident 


Somewhat confident 


O 
©. Not very confident 
©. Notatall confident 


How confident are you about your understanding of and ability to implement the 5E Learning 


Cycle since attending QUEST? 
©. Very confident 


© Somewhat confident 
©. Not very confident 
O 


Not at all confident 
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Please elaborate on your response to the item above with an example about a change in your 
understanding or ability to implement the 5E Learning Cycle. 


Were you a preservice teacher during the 2015 Summer Institute? 
© Yes 


O No 


Discuss something your participation in QUEST has enabled you to do WITHIN your classroom 
that you would not have done previously. 


Discuss something your participation in QUEST has enabled you to do OUTSIDE your classroom 
that you would not have done previously. 


Did you have support from your school/district to implement what you learned in QUEST? 
Please explain. 


In what way were you affected the most by your participation in the QUEST program? Please 
explain. 


Were you able to apply what you learned in QUEST to teaching other content areas? Please 
explain. 


Did you have support from your QUEST colleagues throughout the school year? Please explain. 


What would you have liked to spend more time on during the follow-up PD sessions? 


What would you have liked to spend less time on during the follow-up PD sessions? 


Please describe something you learned in the follow-up PD sessions that contributed to your 
understanding or implementation of science instruction. 
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Preservice Teachers Only: 


Were you able to apply what you learned in QUEST to actually teaching science? If so, in what 
context and how effectively? 


Describe something your participation in QUEST has enabled you to do that you would not have 
done previously. 


What about your participation in the QUEST program affected you the most? Please explain. 


—~ 
| 


educational policy improvement center 


